Abstract

This article discusses recent innovations in how peer review is conducted in light of the various functions journals fulfill in scholarly communities.

Scholarly journals have been in existence for over 340 years. While peer review was less common among early journals, the majority of scientific and scholarly journals implement some level of peer review today. Despite its long history and firm establishment in scholarly communities, peer review has come under increasing scrutiny by scholars (Debate 2006) and even in the lay press (Chang 2006). The debate appears to be largely fueled by the growing impact of electronic dissemination and the use of the Internet in conducting peer review. The Internet has not only reduced the cost and effort of conducting peer review through highly automated Web-based management systems,[1] it has provided a great deal of flexibility in how peer review can be conducted.

Many writers have advocated various forms of “open” peer review. To some extent, these calls for change have focused on eliminating the tradition of blinding the reviewers’ identities. Others have advocated making the full peer-review record public or opening the review process to anyone who wishes to provide comments. There have even been discussions of treating publications as organic documents that evolve over time with a series of versions that change to reflect new information and additional commentary (Bloom 2006).

A number of highly respected journals have begun experimenting with innovative peer-review models. The British Medical Journal did away with blinding in their peer-review process as early as 1999 (Smith 1999) and many of the BioMed Central journals provide open access to the complete review record. For a three-month period starting in June 2006, Nature experimented with posting preprints for public comment in parallel with traditional peer review (Campbell 2006) and the Public Library of Science (PLoS) is in the process of launching a new journal, PLos One that will publish articles almost immediately with minimal screening and allow for public comment.

Although these experiments in peer review are creating a significant amount of discussion, the majority of scholarly journals continue to use traditional methods of peer review in which a selected group of experts whose identity is blinded from the author and public provides feedback to the editor who makes a final publication decision. While it is likely to be a slow process, we appear to be entering an era when the peer review will evolve to take greater advantage of the flexibility offered by the Internet. In my view, this is only one aspect of a much larger transformation of the centuries-old scholarly journal system that is occurring at least in part due to the inherent differences between paper and electronic distribution and, more generally, the flexibility and efficiencies in communication offered by the Internet.

Almost thirteen years ago, Ann Schaffner (1994) wrote a very insightful article discussing the future of scientific journals. She focused on the varied and complex roles journals have played in scientific and scholarly communities in trying to understand the impact of new technology on these journals. If one looks at the debate over how peer review should be done (if at all), arguments often reflect the writer’s or speaker’s focus on a specific function or functions journals play in scholarly communities. It is my belief that we can have a more thoughtful discussion of the role of peer review and the value of the new modes of peer review if we follow Schaffner’s lead in considering the specific roles of journals in scholarly communities when considering how the peer-review process can make best use of the options offered by electronic communication.

The Roles of Journals in Scholarly Communities

Schaffner (1994) identified five distinct, though somewhat overlapping, roles journals play in scholarly communities. As she notes, they could probably be categorized somewhat differently, but for me her scheme makes a lot of sense.

Building a collective knowledge base – Probably the most important role journals play is forming our archive of knowledge. Most people would agree that journals form the most comprehensive, up-to-date, and authoritative archive of information in a given scholarly field. Obviously, the accuracy and quality of the material contained in this archive is of central importance. Peer review serves as one of the most important mechanisms for validating the information contained in these journals.

It generally takes about eighteen months for a peer-reviewed article to go from submission to publication. While this can be reduced to some extent by Web-based peer review and electronic publication, the peer-review and revision process just takes time. Also, the care and effort it takes to develop a publishable manuscript means it can be months between the time research is conducted and the results are even submitted for publication. Additionally, manuscripts are often not accepted the first time they are submitted to a journal and may be resubmitted to several journals before being accepted for publication. The net result is that the information disseminated through peer-reviewed journals is often several years old.

From the perspective of our archive of knowledge, the speed in which the archive is updated, while not trivial, is far less of a concern than accuracy and the quality of the material. I think most people would agree that getting it right through the care and effort of peer review and careful revision, copy editing, and typesetting far outweighs the need for rapid publication when viewing these journals as our archive of knowledge.

Communicating information – Communication among scholars working in a particular field sounds similar to building an achive of knowledge. However, there are important differences. Speed and interactivity are much more important for this role. At the same time, peer review is far less important, as experts in a field are perfectly capable of making their own decisions about the value and accuracy of the information that is being disseminated. Some have argued that without peer review scholars would be inundated with information (Editorial 2005). This does not appear to be the case in the field of physics. The well-known preprint server arXiv.org has as many as 5,000 preprints deposited each month and tens of thousands of people access the site each day, and the system appears to have worked exceedingly well—at least in that field—for over fifteen years.[2]

Journals likely played a much more central role in this type of communication early in their history. With other more efficient means of communication available today, one would think journals would have a fairly limited role in communicating research results among scholars. It is not clear that this is the case. First, the research on informal communication of science and engineering knowledge suggests the modes of communication vary substantially among fields (Faxon Institute 1991). For example, preprint archives such as arXiv.org were quickly embraced by a number of fields but are rarely used in other fields despite concerted attempts by individuals to implement them. (This has been the case in my field of educational research.) The research on informal communication among scientists also suggests that much of what is discussed among scholars turns out to be journal articles (Schaffner 1994). While preprint archives, listservs, and threaded discussions are likely to grow in importance for communication among scholars working in a field, journals clearly appear to be retaining a significant role in this type of communication.

Validating the quality of research – Journals also play a role in maintaining community standards in how research and scholarship are conducted. To some extent, this is done as journals filter what is published and hence disseminated. The effects can also be more subtle. The work of experienced scholars rarely receive harsh reviews. That is not to say they always get their manuscripts published, but they tend to have internalized the norms of the field and know how the research or scholarship should be conducted and described, and are much less likely than novices to be chastised by reviewers.

There is not universal agreement that this is entirely a good thing. Some have argued that this stifles creativity and unnecessarily subjects novice researchers to harsh criticism (Kumashiro 2005).

Distributing rewards – Publication in peer-reviewed journals is one of the major ways scholars are evaluated. Not only is quantity important, but which journals one publishes in is equally if not more important. The roots of this go all the way back to the formation of Philosophical Transactions of the Royal Society of London in the middle of the 17^th century.While tenure decisions were not involved, an important function of that journal was establishing who deserved credit for specific findings or theories. According to Guédon (2001), the journal served almost like a patent office for ideas. By publishing in the journal, scientists or natural philosophers (as they were called at the time) could establish ownership of their intellectual property. Competition among scientists and controversy over credit for discoveries is still an issue today, and journals still fulfill the role of documenting the paternity of intellectual property. This role clearly has expanded as a more general measure of achievement, and peer review—rightly or wrongly—is an important aspect of this role.

Building scientific communities – Journals also act as a means of tying a scholarly community together in a number of ways. A hallmark of a discipline’s coming of age is the establishment of a new journal: in essence, staking out the intellectual territory of the new field. Beyond that, editorials, opinion articles, and letters to the editor often serve as a forum to debate the issues in the discipline. Sometimes they are substantive and sometimes they extend to related areas such as the social implications of findings, funding, or training issues within the field. Journals also commonly serve as a forum for news such as new appointments to major positions or the passing of a well-known member of the scholarly community. While this role may be diminishing to some extent with the variety of communication options available, journals continue to play an important role in forming and maintaining scholarly communities.

Peer Review in Relation to the Roles of Journals

The relevance of peer review clearly varies among the different functions of journals. Peer review is generally seen as vital for the roles of forming an archive of knowledge and distributing rewards. It is also plays a key role in validating the quality of research in a field but, as noted by Kumashiro, may also hamper disseminating new ideas and methods. Peer review is of little value and probably a hindrance in facilitating communication among scholars in a field and is not relevant for the role of building scientific communities.

The value of peer review is based on the assumption that it provides a valid measure of the quality of a manuscript and its adherence to the norms of the field. Its value is also tied to providing feedback so that a manuscript can be improved through revision. These assumptions are largely taken for granted and rarely challenged, yet their validity is open to question.

Jefferson et al. (2002a) performed a systematic review of the literature on the effect of peer review in biomedical journals. They found few controlled studies, and most these were focused on specific editorial practices such as blinding or using checklists in the review process. In a further review (2002b) the authors identified ten studies that looked at the relationship between peer review and the quality of articles. Only one compared peer-reviewed articles with non-peer-reviewed articles, and it had a weak study design. The rest compared various methods of peer review or the change in quality before and after revision based on peer review. Each article used expert ratings based on a different rating scale. None of rating instruments appeared to have been assessed for psychometric quality. Jefferson et al.’s overall conclusion is that there is very little scientifically sound evidence supporting the value of peer review in ensuring the quality of manuscripts—at least in the biomedical sciences.

There is also some evidence that peer review is not necessarily successful in identifying methodological flaws in research articles. Baxt et al. (1998) sent reviewers a fictitious manuscript with clear design and analysis flaws that precluded the results supporting the conclusions of the study that was described. For example, it stated there was random assignment to treatment/control groups but the procedure was clearly not random. The 203 emergency physicians who participated in the study were experienced reviewers for Annals of Emergency Medicine, a well-respected journal in the field. On average the reviewers identified only 34% of the fatal flaws in the manuscript, and 41% of the reviewers indicated the manuscript should be accepted for publication. There is also a multitude of studies demonstrating the high rate of methodological errors in published medical research (Pocock, Hughes, & Lee 1987; Gotzche 1989). Altman (2002) blames this on a variety of causes including the lack of statistical and research design expertise among reviewers. While these examples come from the biomedical sciences, one would think, not to mention hope, that the most sound research practices would be used in such a critical field. Based on my experience as social scientist, methodological errors are common in social science journals as well.

Effective Use of Peer Review

Does the lack of evidence validating the effectiveness of peer review and the fact that flawed research is often published in rigorously peer-reviewed journals suggest that peer review lacks value? I think not. It in many ways peer review parallels our jury system: while flawed, it is the best we have (Jefferson 2006). My appreciation for the process has grown over the eleven years I have been an editor of Medical Education Online (MEO), an open-access peer-reviewed journal in medical education. The value of peer review is not so much as a means of filtering poor manuscripts (though it is helpful to have the backing of several reviewers when faced with an irate author); instead, peer review is valuable as a means of enhancing the quality of what is published. I am continually amazed at the time, effort, and thought that many reviewers put into the review process. The result more often than not is excellent constructive feedback that most authors welcome and use to improve their manuscripts.

Another fact that is sometimes overlooked is that while individual reviewers often miss specific issues in a manuscript, another reviewer often catches the problem. The more reviewers that evaluate a manuscript, the more likely errors will get caught and problems identified. Using the Internet to conduct reviews dramatically reduces the cost and effort involved in peer review and makes it feasible to include a larger number of reviewers per manuscript. Our goal at MEO is to have four to six consultants review each manuscript, and if there are more, even better. By using a fairly open process for selecting reviewers we have found it easy to accomplish this goal. We currently have approximately three hundred reviewers who have volunteered to review manuscripts and have found it relatively easy to add to our review pool as needed.

For peer review to work effectively, the role of an editor is crucial. While most reviews provide valuable feedback, there is a great deal of variability across reviewers in the issues addressed, and the feedback is occasionally contradictory (and once in a while just plain wrong). Someone needs to make sense out of the various reviews and provide a coherent summary of the feedback, and if the manuscript needs revision before publication, a clear set of directions for the author must be given. It is not that the editor is necessarily any wiser or less prone to biases than the individual reviewers, it is just that someone has to take charge so feedback and directions to the author are coherent. Otherwise it drives authors crazy.

At MEO, the review editor provides a feedback letter with the publication decision, a summary of feedback and, if appropriate, a coherent set of issues that must be addressed before publication. In addition, we send back both the comments and ratings from all the reviewers, with the exception of comments the reviewer specified for the editor only. I believe authors appreciate getting all the feedback as well as a clear set of guidelines from the editor for revising their manuscripts. We also send carbon copies of the feedback that is sent to the author to each reviewer. It has been my experience that reviewers almost universally appreciate both learning the final disposition of a manuscript and seeing what the other reviewers had to say about it.

Innovations in Peer Review

As noted, the flexibility and efficiency of communicating via the Internet has made it feasible to experiment with a variety of different models of peer review. The one innovation that has been discussed most in the literature is making the peer-review process more transparent. Traditionally the identities of the peer reviewers are kept confidential, and in many cases manuscripts are masked to remove any identifying information concerning the authors and their institutions. At the most basic level open peer review has consisted of making the identity of the reviewers public or making the identity of the authors known to the reviewers, or both, during the review. This of course is not dependent on electronic publication; review and debate on keeping the identity of reviewers confidential goes back well before the Internet became widely available. This issue has been fairly well researched (Goldbeck-Wood 1999). The evidence suggests that anonymity has little impact on the quality of the review or acceptance rates, but revealing the identity of reviewers may lower the likelihood that someone will volunteer to review (van Rooyen et al. 1999).

Despite the evidence that opening up the peer-review process to public scrutiny does not seem to affect the quality one way or another, the debate has continued, mainly focusing on the ancillary effects of disclosing the identity of the reviewers. Proponents have argued that opening the review process is ethically superior with little or no impact on the process and in fact may encourage more civility in the review process (Godlee 2002; Morrison 2006). The major concerns about open peer review focus on the introduction of personal biases and the potential of retribution in what tends to be a very small world within specific fields. There is a particular concern about the impact on young investigators, who are particularly vulnerable. In a recent survey of both authors and reviewers for Medical Education, a widely read journal in the field, the respondents strongly supported blinding of both authors and reviewers (Regehr and Bordage 2006). There is no clear consensus on whether manuscripts should be blinded, and the arguments on both sides seem persuasive.

Although opening the peer-review process is not contingent on the Internet, the Internet can facilitate an even more transparent review processes. Many of the BioMed Central medical journals use a completely open review process in which not only are the identities of the reviewers and authors public, but the full review record, including all reviewer feedback and subsequent iterations of correspondence in the revision process, is made publicly available with the published manuscript. Complete disclosure of the review record would probably not be feasible with a paper journal for logistical reasons.

A more recent innovation that is contingent on the Internet has been to open the review process to anyone who wishes to comment on a paper. As noted, Nature recently implemented an open review process on a trial basis in parallel to their normal review process, and the Public Library of Science will be implementing PLoS One, a new journal that will use a two-stage process in which a single academic editor performs an initial screen followed by a system for public comment and discussion.

It remains to be seen how successful these public review systems will be. There are a number of less-well-known specialty journals that have implemented review systems based on public comment. Electronic Transactions on Artificial Intelligence (ETAI) provides a hybrid system of peer review (Sandewall 2006). Manuscripts that fit the scope of the journal are immediately posted for public comment for three months, and if there is a continuing discussion the period can be extended. After the discussion period, authors are given an opportunity to revise their manuscripts based on the feedback, and then the manuscript is sent out for external blind review. The reviewers, however, just provide a publish/reject decision without comment, since the manuscript has already received extensive comment from the public. If accepted, articles are usually published within a month.

Atmospheric Chemistry and Physics uses a two-stage approach that includes public comment (Koop and Poschl 2006). After a pre-screening, manuscripts are published as “discussion papers” for public comment for a period of eight weeks. In addition, designated reviewers publish their signed comments along with unsolicited comments from other readers. Authors also are allowed to respond to the comments. In a second stage, the manuscripts are reviewed using traditional review procedures. Accepted articles are published in the main journal. All the discussion papers and comments are also maintained permanently on the site.

These innovative systems that combine public comment with various forms of more traditional review are intriguing and have real merit. They have the potential of doing a significantly better job of balancing the various roles journals play in scientific and scholarly communities. Rapid initial publication of manuscripts with mechanisms for public comment and discussion serve the role of facilitating communication among scholars. At the same time, the use of this public comment along with more traditional peer review prior to final publication provides the same or even possibly a higher level of quality control as traditional peer review. The process of publicly discussing papers also fits well with the roles of fostering scientific communities and disseminating and sustaining standards for conducting research and scholarship.

Over time, we will see if these new approaches to the centuries-old peer-review process add real value. This new era of electronic publishing is still in its infancy and we have much to learn about how best to make use of the new media and communication tools. The fact that our scholarly publication system remained largely intact with only modest changes for over 340 years through huge advancements in science and technology is a testament to how well it has worked. Our challenge over the next decade will be to adapt the system to this new and very different medium, maintaining the components that continue to work well and finding ways to incorporate innovative approaches to communication and evaluation that build upon the capabilities provided by electronic publication.

David Solomon is an educational psychologist who has worked in medical education for nineteen years. He is currently an Associate Professor in the Department of Medicine and the Office of Educational Research and Development in the College of Human Medicine at Michigan State University. His research has been mainly in the areas of performance assessment, specialty choice, and distance learning. In 1996 he started an electronic journal, Medical Education Online, which has grown to be a well-established journal in the field. His other major interest is promoting open-access publishing. He can be reached at dsolomon@msu.edu.

References

Altman, D. G. 2002 Poor-quality medical research: What can journals do? JAMA 287;21:2765-2767. [doi: 10.1001/jama.287.21.2765]

Baxt, W. G., J. F. Waeckerle, J. Berlin, and M. L. Callahm. 1989. Who reviews the reviewers? Feasibility of using fictitious manuscript to evaluate peer reviewer performance. Annals of Emergency Medicine 32:310-317. [doi: 10.1016/S0196-0644(98)70006-X]

Bloom, T. 2006. Systems: Online frontiers of the peer-reviewed literature. In Nature. http://www.nature.com/nature/peerreview/debate/index.html.

Campbell, P. 2006. Nature Peer Review Trial and Debate. In Nature. http://www.nature.com/nature/peerreview/index.html.

Chang, A. 2006. Online journals challenge scientific peer review. Retrieved on November 19, 2006 from http://www.mercurynews.com/mld/mercurynews/news/breaking_news/15655422.htm.

Debate. 2006. “Peer Review” In Nature. http://www.nature.com/nature/peerreview/debate/index.html

Editorial. 2005. “Revolutionized peer review?”. Nature Neuroscience 8; 4:397.

Faxon Institute. 1991. An Examination of Work-related Information Acquisition and Usage among Scientific, Medical and Technical Fields Westwood, Mass. Faxon Company.

Godlee, F. 2002. Making reviewers visible. Openness, accountability and credit. JAMA287; 21:2762-2765. [doi: 10.1001/jama.287.21.2762]

Goldbeck-Wood, S. 1999. Evidence on peer review: scientific quality control or smokescreen? BMJ 318:44-45 (2 January). http://www.bmj.com/cgi/reprint/318/7175/44.

Gotzxche, P. C. 1989. Methodology and overt and hidden bias in reports of 196 double-blinded trials of non-steroidal anti-inflammatory drugs in rheumatoid arthritis. Controlled Clinical Trials, 10:3159.

Guédon, J. 2001. In Oldenburg’s long shadow: Librarians, research scientists, publishers, and the control of scientific publishing. Presentation to the May 2001 meeting of the Association of Research Libraries (ARL), at http://www.arl.org/arl/proceedings/138/guedon.html.

Jefferson, T., P. Alderson, E. Wager, and F. Davidoff. 2002a. Effects of editorial peer review. JAMA June 5, 287; 21:2784-2786.

Jefferson, T, E. Wager, and F. Davidoff. 2002b. Measuring quality of editorial peer review. JAMA June 5, 287; 21:2786-2790.

Jefferson, T. 2006. Quality and value: Models of quality control for scientific research. In Nature . http://www.nature.com/nature/peerreview/debate/nature05031.html.

Koop T., and U. Pöschl. 2006. An open, two-stage peer-review journal. In Nature . http://www.nature.com/nature/peerreview/debate/nature04988.html.

Kumashiro, K. K. 2005. Thinking Collaboratively about the Peer-Review Process for Journal-Article Publication. Harvard Educational Review 75; 3:257-287.

Pocock, S. J., M. D. Hughes, and R. J. Lee. 1987. Statistical problems in the reporting of clinical trials. A survey of three medical journals. New England Journal of Medicine 317:426-432.

Regehr, G., and G. Bordage. 2006. To blind or not to blind? What authors and reviewers prefer. Medical Education 40:832-839. [doi: 10.1111/j.1365-2929.2006.02539.x]

Schafner, A. C. 1994. The future of scientific journals: Lessons from the past. Information Technology and Libraries 13:239-47.

Sandewall, E. 2006. A hybrid system of peer review. In Nature. http://www.nature.com/nature/peerreview/debate/nature04994.html.

Smith, R. 1999 Opening up BMJ peer review. BMJ 1318:4-5 (2 January). http://www.bmj.com/cgi/content/full/318/7175/4.

Van Rooyen S., F. Godlee, S. Evans, N. Black, and R. Smith. 1999. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomized trial. BMJ 318:23-27 (2 January). http://www.bmj.com/cgi/reprint/318/7175/23.