Wikipedia

Who's Afraid of Peer Review?

"Who's Afraid of Peer Review?" is an article written by Science correspondent John Bohannon that describes his investigation of peer review among fee-charging open-access journals. Between January and August 2013, Bohannon submitted fake scientific papers to 304 journals owned by fee-charging open access publishers. The papers, writes Bohannon, "were designed with such grave and obvious scientific flaws that they should have been rejected immediately by editors and peer reviewers", but 60% of the journals accepted them. The article and associated data were published in the 4 October 2013 issue of Science as open access.[2][3]

Letharia vulpina, one of the species of lichen credited with having a cancer-inhibiting molecule in a fake manuscript.[1]

Background

edit

The first fee-charging open access scientific journals began appearing in 2000 with the creation of BioMed Central and then the Public Library of Science. Rather than deriving at least some of their revenue from subscription fees, fee-charging open access journals only charge the authors (or their funders) a publication fee. The published papers are then freely available on the internet. This business model, gold open access, is one of several solutions devised to make open access publishing sustainable.[4] The number of articles published open access, or made freely available after some time behind a paywall (delayed open access), has grown rapidly. A 2013 study showed more than half of the scientific papers published in 2011 were available for free.[5]

In part because of the low barrier to entry into this market, as well as the fast and potentially large return on investment, many so-called "predatory publishers" have created low-quality journals that provide little to no peer review or editorial control, essentially publishing every submitted article as long as the publication fee is paid. Some of these publishers additionally deceive authors about publication fees, use the names of scientists as editors and reviewers without their knowledge, and/or obfuscate the true location and identity of the publishers.[6] The prevalence of these deceptive publishers, and what the scientific community should do about them, has been debated.[7]

Methods

edit

Fake papers

edit

Bohannon used Python to create a "scientific version of Mad Libs".[2][8] The paper's template is "Molecule X from lichen species Y inhibits the growth of cancer cell Z". He created a database of molecules, lichens, and cancer cells to substitute for X, Y, and Z. The data and conclusions were identical in every paper. The authors and their affiliations were also unique and fake. The papers all described the discovery of a new cancer drug extracted from a lichen, but the data did not support that conclusion and the papers had intentionally obvious flaws.[9][10]

Publisher targets

edit

To build a comprehensive list of fee-charging open access publishers, Bohannon relied on two sources: Beall's List of predatory publishers and the Directory of Open Access Journals (DOAJ). After filtering both lists for open access journals published in English, that charge authors a publication fee, and that have at least one medical, biological, or chemical journal, the list of targets included 304 publishers: 167 from the DOAJ, 121 from Beall's list, and 16 that were listed by both. The investigation focused entirely on fee-charging open access journals. Bohannon did not include other types of open access journals or subscription journals for comparison because the turnaround time for reviews in traditional journals is too long. The study consequently makes no claim about the relative quality of the different types of journals.[11]

Results

edit

Acceptance versus rejection

edit

In total, 157 of the journals accepted the paper and 98 rejected it, with the other 49 not having completed their evaluation by the time Bohannon wrote his article.[2] Of the 255 papers that underwent the entire peer review process to acceptance or rejection, about 60% of the final decisions occurred with no sign of actual peer review. For rejections, that may possibly have reflected filtering at the editorial level, but for acceptance can only reflect a flawed process. Only 36 submissions generated review comments recognizing any of the paper's scientific problems. 16 of those 36 papers were nonetheless accepted, in spite of poor to damning reviews. Many of the journals that accepted the paper are published by prestigious institutions and publishing companies, including Elsevier, SAGE, Wolters Kluwer - through its subsidiary Medknow (Journal of Natural Pharmaceuticals), and several universities.[2]

Among those that rejected the paper are journals published by PLOS (PLOS ONE), MDPI (Cancer) and Hindawi (Chemotherapy Research and Practice, ISRN Oncology). The peer review provided by PLOS ONE was reported to be the most rigorous of all, and it was the only journal that identified the paper's ethical problems, for example the lack of documentation of how animals were treated in the creation of the cancer cell lines.[2]

DOAJ versus Beall's list

edit

Among the publishers on Beall's list that completed the review process, 82% accepted the paper. Bohannon stated "the results show that Beall is good at spotting publishers with poor quality control." According to Jeffrey Beall, who created the list, this supports his claim to be identifying "predatory" publishers.[12] However, the remaining 18% of publishers identified by Beall as predatory rejected the fake paper, causing science communicator Phil Davis to state, "That means that Beall is falsely accusing nearly one in five."[13]

Among the DOAJ publishers that completed the review process, 45% accepted the paper.[13] According to a statement published on the DOAJ website, new criteria for inclusion in the DOAJ are being implemented.[14]

Global map of journal fraud

edit

Along with the report, Science published a map that shows the location of publishers, editors, and their bank accounts, color-coded by acceptance or rejection of the paper. The locations were derived from IP address traces within the raw headers of e-mails, WHOIS registrations, and bank invoices for publication fees. India emerged as the world's largest base for fee-charging open-access publishing, with 64 accepting the fatally flawed papers and only 15 rejecting it.[2] The United States is the next largest base, with 29 publishers accepting the paper and 26 rejecting it. In Africa, Nigeria has the largest number, of which 100% accepted the paper.[15]

Responses

edit

Responses from open-access academic publishers

edit

Since the story was released, publishers of three journals have stated that they are shutting them down.[16] The DOAJ is reviewing its list and instituting tighter criteria for inclusion.[17] The Open Access Scholarly Publishers Association (OASPA) formed a committee to investigate the circumstances that led to the acceptance of the fake paper by three of its members.[18] On 11 November 2013, OASPA terminated the membership of two publishers (Dove Medical Press and Hikari Ltd.) who accepted the fake paper. SAGE Publications, which also accepted a fake paper, was put "under review" for 6 months.[19] Sage announced in a statement that it was reviewing the journal that accepted the fake paper, but that it would not shut it down.[20] Sage's membership was reinstated at the end of the review period following changes to the journal's editorial processes.[21]

Responses from the scientific community

edit

Within hours of its publication, the Science investigation came under intense criticism by some supporters of the open-access movement.[22][23]

The first substantial critique was posted by PLOS cofounder Michael Eisen on his blog. "To suggest – as Science (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams. There are deep problems with science publishing. But the way to fix this is not to curtail open-access publishing. It is to fix peer review."[24] Eisen pointed out the irony of a subscription-based journal like Science publishing this report when its own peer review has failed so badly before, as in the 2010 publication of the arsenic DNA paper.

In an exchange between Eisen and Bohannon in a discussion hosted by Peter Suber, director of the Harvard Open Access Project, Eisen criticized the investigation for the bad publicity it generated for the open-access movement.[25] "Your study exclusively targeted open access journals – [which] strongly suggested, whether you meant to suggest this or not, that open access journals are more likely to engage in shoddy peer review and therefore more deserving of scrutiny." Bohannon responded that this critique was equivalent to "shoot[ing] the messenger".[25]

There have also been many statements of support for the investigation,[26][13] and statements of concern about the publishing fraud that it revealed.[27][28][29] The Committee on Publication Ethics has responded that "There is no doubt that this 'sting' raises a number of issues ... though I'd argue they are not necessarily the ones that Science thinks are top priorities."[30]

Implications

edit

Some scientists have discussed a number of options for making peer review more transparent.[31] Doing so would make it harder to maintain a predatory journal that does no peer review, because the record of peer review would be lacking or would need to be faked.[32] Another option is to more rigorously vet journals, for example by further empowering DOAJ or OASPA. DOAJ has recently tightened up its inclusion criteria, with the purpose of serving as a whitelist, very much like Beall's List has been a blacklist.[33]

See also

edit

References

edit
  1. ^ Paper 1 in the data supplement for Bohannon 2013.
  2. ^ a b c d e f Bohannon, John (2013). "Who's Afraid of Peer Review?". Science. 342 (6154): 60–65. Bibcode:2013Sci...342...60B. doi:10.1126/science.342.6154.60. PMID 24092725.
  3. ^ Bohannon, John (2013a). "Data and Documents; Who's Afraid of Peer Review". Science. 342 (6154): 60–65. doi:10.1126/science.342.6154.60. PMID 24092725.
  4. ^ Suber, Peter. "Open Access Overview". Earlham College. Retrieved 21 October 2013.
  5. ^ Van Noorden, Richard (20 August 2013). "Half of 2011 papers now free to read". Nature. 500 (7463): 386–387. Bibcode:2013Natur.500..386V. doi:10.1038/500386a. ISSN 0028-0836. PMID 23969438.
  6. ^ Knox, Richard (3 October 2013). "Some Online Journals Will Publish Fake Science, For A Fee". NPR. Retrieved 20 October 2013.
  7. ^ Butler, Declan (27 March 2013). "Investigating journals: The dark side of publishing". Nature. 495 (7442): 433–435. Bibcode:2013Natur.495..433B. doi:10.1038/495433a. ISSN 0028-0836. PMID 23538810.
  8. ^ Koebler, Jason. "Inside Science Magazine's 'Sting' of Open Access Journals". Motherboard. Retrieved 22 October 2013.
  9. ^ "Science's Sokal moment". The Economist. 5 October 2013. Retrieved 20 October 2013.
  10. ^ Vergano, Dan (3 October 2013). "Fake Cancer Study Spotlights Bogus Science Journals". National Geographic. Archived from the original on 3 October 2013. Retrieved 22 October 2013.
  11. ^ Oransky, Ivan (3 October 2013). "Science reporter spoofs hundreds of open access journals with fake papers". Retraction Watch. Retrieved 22 October 2013.
  12. ^ Beall, Jeffrey. "Science Magazine Conducts Sting Operation on OA Publishers". Scholarly Open Access. Archived from the original on 12 October 2013. Retrieved 22 October 2013.
  13. ^ a b c Davis, Phil (4 October 2013). "Open Access "Sting" Reveals Deception, Missed Opportunities". The Scholarly Kitchen. Retrieved 20 October 2013.
  14. ^ "DOAJ's response to the recent article in Science entitled "Who's Afraid of Peer Review?"". Directory of Open Access Journals. 4 October 2013. Archived from the original on 23 October 2013. Retrieved 22 October 2013.
  15. ^ "Peer review map". Science. AAAS. Retrieved 21 October 2013.
  16. ^ Oransky, Ivan (17 October 2013). "Fallout from Science's publisher sting: Journal closes in Croatia". Retraction Watch.
  17. ^ "Second response to the Bohannon article". Directory of Open Access Journals. 18 October 2013. Archived from the original on 10 November 2013. Retrieved 21 October 2013.
  18. ^ Redhead, Claire (11 November 2013). "OASPA's response to the recent article in Science entitled "Who's Afraid of Peer Review?"". Open Access Scholarly Publishers Association. Retrieved 21 October 2013.
  19. ^ Redhead, Claire (4 October 2013). "OASPA's second statement following the article in Science entitled "Who's Afraid of Peer Review?"". Open Access Scholarly Publishers Association. Retrieved 11 November 2013.
  20. ^ Gamboa, Camille. "Statement by SAGE on the Journal of International Medical Research". Sage. Archived from the original on 14 October 2013. Retrieved 22 November 2013.
  21. ^ Shaffi, Sarah (29 April 2014). "OASPA reinstates Sage membership". The Bookseller. Retrieved 2 June 2014.
  22. ^ Basken, Paul (4 October 2013). "Critics Say Sting on Open-Access Journals Misses Larger Point". The Chronicle of Higher Education. Retrieved 20 October 2013.
  23. ^ Davis, Phil (12 November 2013). "Post Open Access Sting: An Interview With John Bohannon". The Scholarly Kitchen. Retrieved 12 November 2013.
  24. ^ Eisen, Michael. "I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals". Retrieved 21 October 2013.
  25. ^ a b Peter Suber. "New "sting" of weak open-access journals". Archived from the original on 8 October 2013.
  26. ^ Yirka, Bob (4 October 2013). "Research paper publishing sting reveals lax standards of many open-access journals". Phys.org. Retrieved 20 October 2013.
  27. ^ Tatalović, Mićo (10 March 2013). "Sting exposes 'wild west' of open-access publishing". Scidev.net. Retrieved 20 October 2013.
  28. ^ Mudur, G S (3 October 2013). "Dubious journal fear stalks India". The Telegraph. Archived from the original on 16 December 2013. Retrieved 20 October 2013.
  29. ^ Shieber, Stuart (15 October 2013). "Lessons from the faux journal investigation". The Occasional Pamphlet. Harvard Law School. Retrieved 20 October 2013.
  30. ^ "COPE response to Science paper submission of fake paper, by Virginia Barbour, on behalf of COPE council". COPE. 10 October 2013.
  31. ^ "Transparency in peer review". Nature Materials. 10 (2): 81. 2011. Bibcode:2011NatMa..10...81.. doi:10.1038/nmat2952. PMID 21258345.
  32. ^ Bonnie Swoger (26 November 2014). "Is this peer reviewed? Predatory journals and the transparency of peer review". Information Culture. Scientific American.
  33. ^ Van Noorden, R. (2014). "Open-access website gets tough". Nature. 512 (7512): 17. Bibcode:2014Natur.512...17V. doi:10.1038/512017a. PMID 25100463.
edit
Retrieved from "https://en.wiki.x.io/w/index.php?title=Who%27s_Afraid_of_Peer_Review%3F&oldid=1251795953"