Journal of Linguistics and Language Teaching
Volume 7 (2016) Issue 2
The Status of Peer Review in Applied Linguistics Research
K. James Hartshorn (Provo (Utah), USA)
The peer review process is vital to the evaluation of scholarship in every discipline including Applied Linguistics and its related fields. Yet, in many disciplines, the landscape is shifting as longstanding concerns with peer review resurface in a world awash with changing social expectations and advances in technology that provide innovations in the evaluation and dissemination of scholarship. While scholars in many fields are abandoning traditional methods of review, where do scholars in the fields of Applied Linguistics stand amid such change? The present study identifies the collective voice of the field, regarding peer review as currently practiced in contrast to alternatives gaining traction in other fields. Data elicited from journal editors, editorial board members, and reviewers were analyzed to reveal perceptions of the peer review process, the various roles of reviewers, different methods of review as well as numerous strengths, limitations, and suggestions for improvement that could benefit practice.
Key words: Peer review, scholarship, applied linguistics, online publishing
The peer review process is central to the evaluation and dissemination of accepted scholarship in every discipline including Applied Linguistics and its related fields. Though various forms of peer review have been around for centuries, we see assertions of entrenched problems. Horton (2000) represented the views of some scholars when he claimed,
we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong. (Horton 2000: 148)
Though such indictments of peer review are not new, some of the challenges unique to the twenty-first century and the innovations being used to address them are unprecedented. For example, Gould (2013) warns,
We stand at a tipping point. All of our experience with scholarly review and publishing is under assault. (Gould 2013: 1)
Amid the scramble of well-established journals to adapt to ever-changing delivery systems, Gould claims that peer review “faces a challenge to its very existence” and that without appropriate corrective action, “we may find ourselves in a world largely without any standards for publishing.” (Gould 2013: 1)
In addition to longstanding concerns about review processes, recent changes in technology and emerging types of publication have evolved so rapidly that it is vital that we carefully examine the current state of peer review and its implications for the future. For example, we know little about the extent to which those closest to the review process in Applied Linguistics and its related fields share concerns about the status quo. Nor is it clear whether the custodians of this scholarship are aware of emerging forms of evaluation, innovations in dissemination, or growing concerns regarding review in a world clamoring for more social justice, equity, accountability, and transparency. Thus, the purpose of this study is to carefully examine perceptions of peer review in the field of Applied Linguistics with the aim of identifying consensus regarding emerging trends as well as potential strengths, weaknesses, and areas for improvement.
2 Review of Literature
Simply defined, peer review is the process by which a scholar's work is evaluated by expert peers to determine its suitability for publication in an academic journal. An early method of evaluating a manuscript was editorial review, where decisions rested solely with the editor. It was efficient but prone to bias. Though some of today’s processes for review have been around for centuries, systems used by various journals became more refined and widespread in the twentieth century (Shamoo & Resnik 2015). These systems usually involved some form of blinding to preserve anonymity. The most common method of peer review is the single-blind method, where the author is known to the reviewers, but the reviewers are not known to the author (Ware 2008). Other types of review include the double-blind method, where neither reviewers nor authors know the others’ identities. Less common is the triple-blind method, where editors do not know the author’s identity in the initial stages of the review process (e.g. Gould, 2013).
2.1 Benefits of Peer Review
Peer review has been perpetuated because of its apparent benefits. For example, Shatz (2004) notes “Peer review serves the academic community by controlling the flow of ideas and enhancing the quality of scholarly work” (Shatz 2004: 30). He also observes:
It motivates scholars to produce their best, provides feedback that substantially improves work which is submitted, and enables scholars to identify products they will find worth reading. (Shatz 2004: 30)
Similarly, Ware (2008) suggests that all the key stakeholders benefit from peer review. For example, “editors are supported in their decisions by the views of experts,” “authors benefit from the assistance offered by reviewers,” “readers benefit because of the filter that peer review provides,” and “even reviewers…benefit” with professional development and early access to research (Ware 2008: 12).
2.2 Challenges Associated with Peer Review
Despite these long-held benefits, criticisms of peer review have emerged, based on years of experience and empirical study. For example, Smith (2010) claims, “we have no convincing evidence of its benefits but a lot of evidence of its flaws” (Smith 2010: 1). Many scholars agree with such sentiments. For instance, in a presentation to the American Psychological Association, Schmelkin (cited in Shatz, 2004) summarized studies suggesting that peer review is prone to harmful bias and incompetence, it lacks scientific evidence for its benefits, the process is slow and costly, it produces many papers with appalling flaws, it often misses misconduct from authors while allowing unscrupulous reviewers to steal ideas or to slow or stop progress toward publication, it is subject to may political pressures, and in the guise of anonymity, it often produces reviews that are caustic, arrogant, or irresponsible.
Though many concerns about the peer review process have existed since the beginnings of peer review itself, there are new complexities in the twenty-first century that might be added to the list. For example, in an attempt to adjust for problems associated with bias, incompetence, and misconduct (e.g. Bohannon 2013, Bosch 2014, Clair 2014, Macdonald 2015, Resnik, Gutierrez-Ford & Peddada 2008, Rothwell & Martyn 2000, Thurner & Hanel 2011, Triggle & Triggle 2007, Tsang 2013), some scholars have advocated replacing blind reviews with a more transparent practice. Some scholars agree with editors such as Smith (2006, who recommended that we “open up the whole process and conduct it in real time on the web.” “Peer review,” he continues, “would then be transformed from a black box into an open scientific discourse” (Smith 2006: 181).
2.3 The Effects of Open Review
Interpreting the actual overall benefits of open review, however, may be difficult. Just a few examples may suffice so as to support this point. Jadad, Moore, Carroll & Jenkins (1996) found that blinding produced greater consistency compared to open reviews. However, Van Rooyen, Godlee, Evans, Black & Smith (1999) found no difference in an open review format in terms of review quality, recommendations for publication, or the time required to complete the reviews. However, they did observe that reviewers are less willing to review in an open format. This lack of willingness to participate in open reviews has also been observed by others (e.g. Ware 2008). On the other hand, in a study conducted by Walsh, Rooney, Appleby & Wilkinson (2000), it was found that signed reviews were of greater quality, more polite, and more likely to recommend publication though they took longer to complete. Some of these findings seem consistent with those of Kowalczuk, Dudbridge, Nanda, Harriman & Moylan (2013), who found that open reviews were of greater overall quality, provided better feedback on methods, were more constructive and provided more evidence to substantiate reviewer claims:
2.4 Preprint Servers and Quasi-Review
In addition to various forms of open and interactive review are preprint servers and quasi-reviews. Preprint servers such as ArXiv, Nature Precedings, Math Prepreints, and Cogprints make manuscripts available after a quick check of minimal standards and credentials but without formal reviews. There also are open access venues with a “non-selective review process, which consider only the scientific quality” of a manuscript rather than its “importance” or “novelty” (Walker & Rocha da Silver 2015: 2). Because of such developments, the evaluation and dissemination of academic work in every field are awash in change. Rather than ignore these trends, we should confront, analyze, and seek to understand them and their relevance in our various contexts. Some adjustments may benefit processes associated with reviewing and sharing quality scholarship while others may undermine them.
2.5 Research Questions
Although different approaches to the evaluation of scholarship may be appropriate within different disciplines, there is a clear and present need for scholars within the fields of Applied Linguistics to contribute to the collective voice as the larger field confronts many old and some new challenges on the shifting landscape of peer review. Innovations or adjustments to peer review, if any, should be made by design rather than by default. Moreover, input from a variety of stakeholders needs to be considered in order for findings to be representative. Those closest to the review process will include editors, editorial board members, reviewers, and authors. Such roles, however, are far from exclusive. Invariably, editors also function as reviewers, and both editors and reviewers continue to function as authors. Thus, in an attempt to provide additional insight for such custodians of scholarship within the fields of Applied Linguistics, the following research questions were formed for editors, editorial board members, and reviewers:
1. How satisfied are scholars with the current peer review process?
2. Do peer review processes need to improve as online and self-publication proliferates?
3. What are the primary roles of the reviewer? Are some roles more important than others?
4. How do experts feel about various review methods and processes?
5. What are the strengths and weaknesses with current peer review processes?
6. What suggestions, if any, do scholars have for improving review processes?
Regardless of what the findings may indicate, the answers to such questions should be of great value at a time when many fields are reevaluating their approaches to the review process.
In order to answer the above research questions, a study was designed to elicit self-reported data from respondents with substantial experience as editors, editorial board members, and reviewers. Though critics may be leery of self-reported data over concerns that participants may not provide accurate responses, scholars such as Chan (2009: 326), assert that self-reporting is not only “justifiable” but “necessary” for particular constructs that might not be accessible any other way. The survey instrument was designed to elicit data from objective and open-ended items in order to present quantitative analyses along with respondent commentary.
3.2.1 Identifying Respondents
The intent was to elicit responses from as many international editors, editorial board members, and reviewers in fields related to Applied Linguistics as possible. Potential respondents came from
With extensive overlap of scholars across multiple journals and multiple roles, the focus was on the individuals themselves more than the journals. Nevertheless, specific individuals were only targeted if they had a clear connection to Applied Linguistics and its related fields rather than being limited to narrow specializations in areas such as Psychology, Education or anthropological, historical, forensic, or evolutionary linguistics.
3.2.2 Response Rate
E-mail addresses were collected for editors, board members, and reviewers, where available. Ethics approval was obtained and the survey was constructed and piloted on a local level to identify areas for improvement. The refined instrument (see Appendix) was sent to 760 individuals, based on the best available contact information. The survey delivery software reported that the email was opened by 473 individuals and that the survey was started by 348 persons. Ultimately, 261 respondents completed the survey. This means that 55.2% of the total number of 473 individuals who had been contacted confirmed to have received the email.
3.2.3 Respondent Demographics
This study examined data from three types of respondents, including journal editors, those serving on editorial boards, and reviewers, all of whom were also authors. Since these rolls are not mutually exclusive, definitions were provided. For the purposes of this study, all editors and section editors were considered to be editors. Board members were defined as those serving on an editorial board but not in the capacity of an editor. Reviewers were defined as those who review for journals but who do not serve in the capacity of editor or board member.
This section identifies relevant similarities and differences among journal editors, board members, and reviewers. Table 1 shows that more than three quarters of the respondents serve as editors and that board members and reviewers made up just over a tenth of the respondents each. The table also indicates one of six broad regions in which these scholars do most of their work. While most respondents work in North America and Europe, just over a quarter work in Asia and Oceania. Less than 2% of the respondents reported working in other locations:
Table 1: Regions in which Respondents Work
Table 2 presents the respondents’ years of service in the roles of editor, member of an editorial board, reviewer, and author. The table includes the mean number of years respondents have served in each role (M) along with standard deviations (SD). There were some notable differences among those who were defined by these various roles. For example, there was a statistically significant difference in the number of years of experience as a reviewer across roles (F(2,245)=8.992, p<.001). As might be expected, editors had more experience reviewing than those defined only as reviewers (p<.001, d= 1.074). Board members also had more years of service compared with those defined only as reviewers (p=.012, d=.933). Similarly, differences were observed across roles in number of years as an author (F(2,242)=8.785, p<.001). Those defined as reviewers had significantly fewer years as an author compared to editors, (p<.001, d=1.056), and fewer years compared to board members (p=.014, d=1.030):
Table 2: Mean Years of Service within Each Roll
Additional demographic information provided in Table 3 presents the total mean number of journals that respondents have worked with as an editor, board member, or reviewer. It also includes the respondents’ average number of publications annually. The mean number of journals with whom respondents worked differed significantly (F(2,257)=4.087, p=.018). For example, reviewers reported working with fewer journals compared with editors (p=.013, d=.575). However, there was no meaningful difference among respondents with varied roles in terms of the mean number of annual publications (F(2,251)=2.466, p=.087):
Table 3: Mean Number of Journals and Annual Publications