INTRODUCTION
Peer review refers to the evaluation of a manuscript by an author’s peers, who are doctors and/or scientists belonging to the same area of research, clinical specialisation or subspecialisation. Peer reviewers aim to provide a critical, independent and unbiased assessment of the submitted manuscripts, and peer review is regarded as a key extension of the scientific process.(1) Therefore, peer reviewing requires a community of experts in a defined field who are qualified, willing and able to perform impartial reviews. As it may be difficult for researchers and authors to detect every mistake or flaw in their own work, particularly if complicated, having the work scrutinised afresh by external expert peers increases the probability that any weaknesses will be identified and improved upon prior to publication.
Peer review greatly aids journal editors in deciding whether a submitted manuscript is suitable for publication, helping to fulfil their important gatekeeping function in determining what goes into the pool of knowledge. Peer review has been adopted by all major medical journals, as it is widely recognised as crucial for assessing the reliability of scientific research and maintaining the journal’s reputation. Working together, the partnership of authors, peer reviewers and editors/publishers brings the products of research and scientific thought to fruition as a published article.
ORIGINS OF PEER REVIEW
Peer review did not always exist. Philosophical Transactions, the journal of the Royal Society of London, was founded in 1665 and is regarded as the world’s first and longest-running scientific periodical. The inaugural editor, Henry Oldenburg, not only decided on the contents of each issue, but also published the journal at his own expense!(2) It was common practice in the first few centuries of scientific publishing for the journal editor to make the sole decision regarding what was published.
In 1752, when the Royal Society took over the financial responsibility for Philosophical Transactions, a standing Committee of Papers was formed. Any paper read before the Society was automatically considered by the Committee of Papers, which was composed of a gentlemanly group of Society members. Therefore, any communication presented to the Society was an implied submission to be considered for publication.(2) Papers that successfully got through this internal peer review process were published.
The honour of introducing peer review to scholarly publication, however, is acknowledged to go to Medical Essays and Observations, a collection of peer-reviewed medical articles published by the Royal Society of Edinburgh in 1731.(3) The peer review system evolved slowly and gradually thereafter. The practice of appointing external peer reviewers became more widespread in the middle of the 20th century. For example, leading journals such as Science and the Journal of the American Medical Association (JAMA) did not use external reviewers until after 1940,(3) while Nature formally introduced peer review only in 1967 and the Lancet implemented peer review in 1976.(4)
Peer review was not considered practical until the invention and more widespread availability of the photocopier from the 1960s onwards, which allowed multiple copies of manuscripts to be mailed to external reviewers without the risk of losing the submitted original material. In the subsequent decades, the marked growth in medical research, occurring in tandem with increasing journal page space limitations, required editors to be more discriminating in selecting articles for publication. This resulted in greater recognition of the importance of peer review as a means to evaluate the quality of submissions.(5)
MODELS AND VARIANTS
Many variations of peer preview processes, formats and models exist among journals. Going into detail about the different journal in-house processes and formats is beyond the scope of this editorial. Regardless of the peer review model, traditionally, the journal editorial team will approach potential peer reviewers, and those who are agreeable will be required to submit a written review by a given deadline. The models of peer review can be broadly classified into blinded and open, with further sub-classifications.
Blinded peer review
The traditional peer review model is blinded: either single-blinded or double-blinded. The single-blinded review model has two further subtypes. In the first subtype, the authors know the names of the reviewers but the reviewers do not know the identity of the authors. This subtype is not common and can appear in the form of an ‘author-guided review’, in which the journal allows author selection of reviewers. As this system is meant to protect against potential reviewers who are biased (e.g. professional rivalry), authors can provide the names of certain reviewers for exclusion. Other advantages are that authors can choose the most appropriate reviewers for their topic, it is easier for editors to find relevant expertise for very specialised fields, and the time taken to review can be reduced. A counterargument is that this sort of peer review distorts the scientific process. There is also potential for reviewer fraud (discussed later).
In the second subtype of the single-blinded review model, the reviewers know the names of the authors, who are blinded to the reviewer’s identity. This subtype is the more common of the two single-blinded models. In this subtype, reviewer anonymity protects the reviewers and the review process. However, as reviewers may be influenced by knowing who the authors are, such as being unduly harsh on rivals or overawed by ‘big name’ authors, the review quality may be compromised. There is a further subvariant where the reviewers are given the option of signing off their reports, such that their identities are eventually made known to the author.
In the double-blinded review model, neither the authors nor the reviewers know each other’s identities. This format is currently used by most medical journals. In principle, this model solves the problem of conflict of interest. It negates the effect of personal dislikes or animosity by reviewers, settling of personal vendettas, as well as the effect of ‘big name’ authors. Possible reviewer bias is removed in double-blinded reviews, which are, hence, favourable to younger researchers, women, minorities/foreigners, lesser-known institutions and non-traditional centres in developing countries. This review model also protects reviewers and the process of review.
However, in small fields or communities with a limited pool of suitable reviewers, it may be difficult to hide author identity despite the best blinding efforts. It is also time-consuming for the editorial office to perform a proper blinding, as author, institutional and location identity may appear in many sections of the manuscript, such as the Materials and Methods, Acknowledgements and References sections. In triple-blinded peer review, the handling editor, reviewer and author are all anonymous to each other. In double- or triple-blinded peer review, the potential for reviewer misconduct exists, such as the temptation for reviewers to plagiarise the author’s (still unpublished) ideas or data.
There is another variant called transparent peer review, where the reviewer reports are published together with the article, but the reviewers are not named. This model aims at increasing the transparency of the review process, so that readers can see the reviewers’ comments and also assess for themselves the quality of the review. In some journals, the editorial comments and correspondence also accompany the article, which provides further transparency to readers about the basis for editorial decisions.
Open peer review
Open peer review aims to overcome the shortcomings of the traditional closed peer review model. Just as peer review became practical only after photocopies were widely available, open review in its current form was possible only after the World Wide Web matured and transformed into Web 2.0.(5) Web 2.0 allows participation, collaboration and information sharing, which are factors ideal for facilitating open peer review. The Internet enables the whole review process to be conducted online openly. The First International Congress on Peer Review in Biomedical Publication, hosted by JAMA in 1989, was regarded as an instrumental forum in promoting open peer review.
In the 1990s, several leading journals started experimenting with hybrid peer review, trying out open peer reviews in parallel with traditional closed reviews. In one of the early studies, 56 research articles accepted for publication in the Medical Journal of Australia (MJA) were published online together with the peer reviewers’ reports. Readers were allowed to submit their comments and the authors could then further amend their articles, before print publication of the article. The conclusion then was that the process had only modest benefits for authors, editors and readers.(6) Initial evidence of early trials with open peer review was mixed.(7-11)
In 1999, the open access Journal of Medical Internet Research was launched, which, from its inception, decided to publish the names of the reviewers at the end of each article. In 1999, the British Medical Journal (BMJ) started an open peer review system that revealed reviewers’ identities to the authors.(12) In 2000, medical journals in the open access BMC series published by BioMed Central were launched using open peer review, with the reviewers’ names included with the peer review reports that were published alongside the accepted articles.
Under the open model of peer review, the authors and reviewers are known to each other. Open reviews can be further subdivided into pre- and post-publication subtypes. In pre-publication peer review, following initial assessment by the editor, the manuscript is sent for open peer review. Upon acceptance, all previous versions of the manuscript are made available online, together with the named reviewers’ comments and authors’ replies to these comments. Reviewers also have the option of making confidential comments to the editor, particularly if they have serious concerns about a manuscript, such as possible plagiarism or fraud. This review model has been adopted by BMJ Open and journals in the BMC stable.
Another variant of the open pre-publication peer review process involves uploading of the manuscripts to a preprint server by the author(s) or editorial office, allowing any number of researchers to read and comment on the manuscript (i.e. open participation by the community). This model provides opportunity for a public and open interchange of thoughts and responses between authors and reviewers. A ‘decoupled review’, where the review is facilitated by a different organisation rather than the venue of publication, can also be conducted on an open platform. A further subvariant – the sequential review process – has been proposed, beginning with the acceptance of a preprint by an e-print server, followed by revision on the basis of comments received publicly or privately, and the solicitation of selected e-prints for commissioned review.(13) Seven core traits of open peer review have been identified.(14)
In post-publication open peer review, the peer review process continues after the article has been published and read. Traditionally, this takes the form of the time-honoured ‘letter to the editor’. Currently, some journals host an online peer forum, where commentaries from readers may be published together with the paper commented upon. This is particularly applicable for journals that e-publish papers online ahead of print. Facebook has also been used successfully as a means of post-publication open peer review by journals such as the New England Journal of Medicine. Some journals have blogs for this purpose, such as the American Journal of Neuroradiology. The BMJ’s ‘rapid responses’ enable readers to submit electronic comments to the editor to be posted online initially and possibly published online after editing.
Independent websites that feature post-publication peer reviews also exist, where readers can make comments of praise or criticism after a paper has been published. Launched in 2012, PubPeer is one such website, where registered users are given the choice between leaving signed comments under their real name or remaining anonymous. Subsequently, PubPeer introduced an option to leave comments anonymously and without registration. Such comments are shown as ‘unregistered’, and quickly became the most popular option.(15) This site has also served as a whistle-blowing platform of sorts, helping to uncover scientific fraud and leading to subsequent paper retractions by journals.(16-18)
F1000Prime is a subscription-based article recommendation and literature discovery service that specialises in post-publication rating of articles in biology and medicine. F1000Prime uses individual scores from their extensive pool of peer reviewers to calculate the total scores for each article, which are used to rank articles. Article recommendations consist of star ratings, one or more tags marking articles as, for example, ‘controversial’ or ‘changes clinical practice’, a brief review, reasons for recommendation and its potential implications.(19)
The advantages of open peer review include speed, transparency and accountability of the review process. This form of review removes the perceived unfairness that the authors’ work should be judged by reviewers ‘hiding’ behind anonymity. As they are named, reviewers receive public recognition and, hence, get rewarded for the important work that they do. Named reviewers are less likely to be biased and more likely to put in their best effort, producing better quality reviews with constructive criticism. By contributing quality reviews in a timely manner, young researchers have the opportunity to display their expertise and advance their careers in public, further incentivising them to keep peer reviewing.
Given the growing trend of ‘science denialism’, having peer review conducted openly and transparently on the Internet may be a useful way to re-establish bonds of trust and confidence in science in the minds of the public.(20) Open peer review enables easier detection of any reviewer conflict of interest by readers. Finally, for early career researchers, being able to observe the peer review process and outcomes is, in itself, a valuable educational exercise.
Several drawbacks of open peer review exist. Because it is a completely voluntary and unsolicited activity, it is sometimes difficult to get enough reviewers, leading to delayed publication. This is a weakness of the open pre-publication peer review variant model, where the manuscripts are uploaded to a preprint server for public scrutiny. Reviewers will be more reluctant to be too hard or harsh on manuscripts if they know that their names will be linked to the report. Reviewers will not want to offend ‘big name’ authors who, for example, evaluate faculty track appointments and promotions; organise conferences and invite speakers; edit journals; and hold influential positions in major learned bodies such as universities, healthcare institutions and professional societies. Instead, reviewers may be tempted to submit favourable reports with unjustified false praise in order to get into the good books of ‘big name’ authors, or because they fear vindictive authors.
Three leading medical communication organisations – the American Medical Writers Association, European Medical Writers Association and International Society for Medical Publication Professionals – recently published a joint statement raising concerns about the dangers of inadequate pre-publication peer review in the rush to get research data released, particularly in the form of preprints. These organisations reiterated that medical research without rigorous pre-publication review may undermine public trust in medical science, and suggested some recommendations aimed at protecting the integrity of published medical research.(21)
The hybrid peer review system incorporates external open peer review, in addition to and following traditional double-blinded closed review. This model is favoured by open reviewers, as they feel that they are not responsible for acceptance or rejection; at the same time, the original blinded reviewers remain anonymous.(5) The disadvantages of this model are increased time required to complete the review cycle and added expenses. A modified hybrid model has been adopted by the Proceedings of the National Academy of Sciences, wherein open reviews take place after acceptance and posting of articles, similar to the MJA trial conducted back in 1996–1997.(6)
Patient peer review
A fairly recent development is the concept of recruiting patients, patient advocates and carers as ‘patient peer reviewers’. The pioneering patient peer review journal, Journal of Participatory Medicine, has had patients and health professionals actively collaborating as full partners since 2009. The BMJ’s launch of its strategy to advance a ‘patient revolution’ in healthcare in 2013(22) led to the ‘partnering with patients’ initiative in 2014.(23) In addition to the journal’s conventional academic reviewers, formal feedback is obtained from patient peer reviewers on the way a study is presented. Evaluation points include whether the issues discussed in a study are relevant to patients, whether challenges faced by patients are highlighted and whether the intervention or treatment introduced has potential benefits for patients.(23)
The BMJ’s key changes included requiring authors to document whether and how they have involved patients in setting the research question, study design and implementation, and dissemination of results; injection of patient reviewers; invitation of patients as sole authors for some commentaries and editorials; and appointing patients to the editorial board.(23) Two other journals that have incorporated patients as peer reviewers as well as editorial board members are the Patient Experience Journal and BMC Research Involvement and Engagement, launched in 2014 and 2015, respectively.
The advantages of having patient peer reviewers are: the inclusion and empowerment of patients who are essential stakeholders in the clinical research process; introduction of a publication model that is co-produced by every stakeholder (including patients, patient advocates and carers); and creation of a new talent pool of peer reviewers. At present, however, the concept of patient peer review is still not fully developed. Concerns include tokenism, failure to systematically collect and use data on patients’ experience and care, inability to get the ‘authentic voice’ of patients heard instead of just that of the ‘articulate minority’, and issues with transparency and commercial influences.(23,24)
CONTROVERSIES
Dr Richard Smith, the former editor of the BMJ, called peer review “slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud”.(12) A damming statement indeed! Articles with eye-catching titles from reputable periodicals, such as: Bioscience: ‘Is peer review a game of chance?’;(25) The New York Times: ‘For sciences’ gatekeepers, a credibility gap’;(26) and Time magazine: ‘Is the peer review process for scientific papers broken’?(27) serve to add more fuel to the peer review fire.
There have been stories of significant papers, some written by Nobel prize laureates, that were rejected by top journals such as Science and Nature following peer review.(28-30) By contrast, many seminal works did not even undergo peer review, including Charles Darwin’s theory of evolution, which appeared in his book On the Origin of Species in 1859, Albert Einstein’s 1905 paper on relativity, and James Watson and Francis Crick’s 1953 letter on the structure of DNA.(26,30) These examples amplify arguments against peer review, which generally come in two categories: that the system (a) wrongly rejects scientifically valid papers and (b) wrongly accepts scientifically flawed papers.(31)
Journals assume that the submitted manuscript has been honestly prepared and the results have been analysed as meticulously as possible by the author. The reviewers make their judgement based on the methodology, data, analyses and conclusions contained in the manuscript. This process fails when a peer-reviewed article contains fundamental errors that undermine at least one of its main conclusions but are not identified by the peer reviewers. Many journals have no procedure to deal with peer review failures beyond publishing letters to the editor.(32)
A famous 1998 experiment on peer review of a fictitious manuscript found that peer reviewers failed to detect a number of manuscript errors and that the majority of reviewers did not notice that the conclusions of the paper were unsupported by its results.(33) Similar findings were reported by another study, where peer reviewers detected an average of only two out of eight ‘areas of weakness’ introduced into a modified manuscript.(8)
The peer review system is not designed to detect fraud, although peer reviewers occasionally manage to do so. Clever fraud is usually uncovered by researchers who have access to the raw data or by other researchers who try to replicate this work later on following publication. When a paper is published with fraudulent or irreproducible data that is subsequently discovered, the paper may be retracted. Journals can report authors to their institutions, but individual authors are expected to take full responsibility and accountability for their offences.(34)
Some journals allow authors to select reviewers, the so-called ‘author-guided review’. Unfortunately, over the recent few years, instances of fraudulent peer review have surfaced. In 2015, fake reviews were uncovered in 21 papers submitted to the Journal of the Renin-Angiotensin-Aldosterone System (JRAAS). The journal discovered that although the listed reviewers of the 21 papers were real people, they had never submitted reports. Fabricated reviewer accounts, consisting of genuine names with falsified email addresses, had been set up in the ScholarOne manuscript processing and peer review system, and selected as recommended reviewers by the authors. All these submissions have since been either retracted or rejected. JRAAS has discontinued author nomination of reviewers.(35)
Similar incidents of manipulated peer review were discovered in 2015, resulting in the BMC retracting 43 articles from several of their journals(36) and Springer retracting 64 papers from ten journals.(37) In response to fake peer review, some publishers, such as BMC and PLoS, have ended the practice of author-suggested reviewers.(37) Becoming aware of systematic and inappropriate attempts to manipulate the peer review processes of several journals across different publishers, the Committee on Publication Ethics (COPE) has issued a statement warning of this fraudulent practice. COPE has also stated that these manipulations appear to have been orchestrated by a number of third-party agencies offering such services to authors in return for a fee.(38)
There is also the growing concern of predatory journals, which take advantage of the trend of charging authors an article submission or processing fee for publication – a practice commonly adopted by open access journals and increasingly employed by some publishers. However, unlike these legitimate entities, predatory journals do not meet the expected scholarly standards, and there is almost always a lack of or negligible peer review. Not only do the authors suffer by not receiving constructive feedback that comes with a robust peer review system, but a great disservice is also done to the readers of such journals.(39)
Anglocentrism has been identified as another limitation of peer review. Despite the rapid growth in global research, a large proportion of the gatekeepers of major journals (i.e. editors and reviewers) are still native English speakers.(40) Reviewers judge the quality of the writing differently depending on their own first language, and tend to consider any manuscript that does not satisfy their personal preferences for ‘good scientific English style’ to be badly written.(41) A solution is the recruitment of a wider sample of reviewers reflective of cultural and linguistic diversity, with varied experiences in healthcare and health research, and hailing from non-Anglophone countries.(40)
THE WAY FORWARD
In 2011, the United Kingdom House of Commons commissioned a report on the state of peer review in scientific publications, which concluded that peer review “is crucial to the reputation and reliability of scientific research”. The report acknowledged that many scientists believe that the system stifles innovation, and peer review requires subjective judgements that may result in errors.(42) It is part of the scientific thought process that research findings, especially if novel or unconventional, will always be questioned – either before publication by the editor and/or reviewers or after publication by readers and other researchers.
The key is for peer reviewers to be well-trained, open-minded and diligent in their evaluations, which are performed to the best of their knowledge and experience. What may be deemed eccentric or non-conformist by the scientific community today may turn out to be mainstream orthodoxy in the future. Peer reviewers still have a useful role in filtering out ideas, scientific methods and conclusions that are downright dangerous or too divergent from the current scientific or medical practice norms.
Currently, editors rely on reports by peer reviewers, which are based on the information submitted by the authors. When warranted, it is good practice for editors and reviewers to ask authors for more information, including the examination of raw data. The peer review process may also be made more robust by conducting audits of reviewer reports, similar to those used for quality control in medical practice. As peer reviewers are a very valuable resource, there are also good grounds for journals to find innovative ways to recognise, retain and reward good peer reviewers.(43)
The inaugural Publons Global State of Peer Review, published in 2018, found that authors preferred to submit their manuscripts to a journal that adopted blinded peer review rather than one with open peer review. Moreover, peer reviewers were more likely to agree to review for a journal with a blinded peer review policy rather than one using open review.(44) In a recent survey of manuscripts submitted to 25 Nature-branded journals, authors were more likely to choose double-blinded peer review if they submitted their manuscripts to more prestigious journals, were affiliated to less prestigious institutions or were from certain specific countries, including China, India and South Korea.(45)
Other studies have shown that double-blinded peer review results in an increase in female first-authored papers(46) and negates the advantages of famous authors and high-prestige institutions.(47) From the collectivist cultural perspective, double-blinded peer review remains appropriate and should be considered a valid model in practice, particularly in Asia.(48) Finally, there has been a recent proposal to decouple peer review – and with it, scholarly communication – from commercial entities and journals, perhaps enabling a return of peer review to the core principles upon which it was founded as a community-based process.(49)
SUMMARY
Currently, peer review is still adopted by all major medical journals, despite being acknowledged to be less than perfect. This process is still considered the best way to assess the quality of the submitted scientific material. Several variations of peer review models exist, with their own advantages and disadvantages. Although open peer review has attracted much attention over the past two decades or so, there seems to be a resurgent appreciation of the merits of double-blinded peer review. Editors must have oversight of and ensure the robustness and quality assurance of their journal’s peer review process. Peer review should be recognised as an altruistic service provided to the scientific community and a key component in the process of translating research and scientific thought to publication.