News:

Also, i dont think discordia attracts any more sociopaths than say, atheism or satanism.

Main Menu

Journalist submits fake paper, passes peer review.

Started by Kai, October 13, 2013, 01:05:29 AM

Previous topic - Next topic

Kai

http://www.sciencemag.org/content/342/6154/60.full

QuoteI know [that it should have been rejected] because I wrote the paper. Ocorrafoo Cobange does not exist, nor does the Wassee Institute of Medicine. Over the past 10 months, I have submitted 304 versions of the wonder drug paper to open-access journals. More than half of the journals accepted the paper, failing to notice its fatal flaws. Beyond that headline result, the data from this sting operation reveal the contours of an emerging Wild West in academic publishing.

QuoteThe paper took this form: Molecule X from lichen species Y inhibits the growth of cancer cell Z. To substitute for those variables, I created a database of molecules, lichens, and cancer cell lines and wrote a computer program to generate hundreds of unique papers. Other than those differences, the scientific content of each paper is identical.

The fictitious authors are affiliated with fictitious African institutions. I generated the authors, such as Ocorrafoo M. L. Cobange, by randomly permuting African first and last names harvested from online databases, and then randomly adding middle initials. For the affiliations, such as the Wassee Institute of Medicine, I randomly combined Swahili words and African names with generic institutional words and African capital cities. My hope was that using developing world authors and institutions would arouse less suspicion if a curious editor were to find nothing about them on the Internet.

QuoteThere are numerous red flags in the papers, with the most obvious in the first data plot. The graph's caption claims that it shows a "dose-dependent" effect on cell growth—the paper's linchpin result—but the data clearly show the opposite. The molecule is tested across a staggering five orders of magnitude of concentrations, all the way down to picomolar levels. And yet, the effect on the cells is modest and identical at every concentration.

One glance at the paper's Materials & Methods section reveals the obvious explanation for this outlandish result. The molecule was dissolved in a buffer containing an unusually large amount of ethanol. The control group of cells should have been treated with the same buffer, but they were not. Thus, the molecule's observed "effect" on cell growth is nothing more than the well-known cytotoxic effect of alcohol.

In short, of the 255 journals that the fake paper was submitted to (not counting the derelict journals), 157 of them passed peer review.

Michael Eisen points out the hilarity of this expose being published in Science: http://www.michaeleisen.org/blog/?p=1439

QuoteMy sting exposed the seedy underside of "subscription-based" scholarly publishing, where some journals routinely lower their standards – in this case by sending the paper to reviewers they knew would be sympathetic - in order to pump up their impact factor and increase subscription revenue. Maybe there are journals out there who do subscription-based publishing right – but my experience should serve as a warning to people thinking about submitting their work to Science and other journals like it.

OK – this isn't exactly what happened. I didn't actually write the paper. Far more frighteningly, it was a real paper that contained all of the flaws described above that was actually accepted, and ultimately published, by Science.

That's right, the arsenic-eating bacteria paper from last year was published by Science, a big name, "closed-access" journal. He goes on to argue:

QuoteBut it's nuts to construe this as a problem unique to open access publishing, if for no other reason than the study, didn't do the control of submitting the same paper to subscription-based publishers (UPDATE: The author, Bohannon emailed to say that, while his original intention was to look at all journals, practical constraints limited him to OA journals, and that Science played no role in this decision). We obviously don't know what subscription journals would have done with this paper, but there is every reason to believe that a large number of them would also have accepted the paper (it has many features in common with the arsenic DNA paper afterall). Like OA journals, a lot of subscription-based journals have businesses based on accepting lots of papers with little regard to their importance or even validity. When Elsevier and other big commercial publishers pitch their "big deal", the main thing they push is the number of papers they have in their collection. And one look at many of their journals shows that they also will accept almost anything.

None of this will stop anti-open access campaigners  (hello Scholarly Kitchen) from spinning this as a repudiation for enabling fraud. But the real story is that a fair number of journals who actually carried out peer review still accepted the paper, and the lesson people should take home from this story not that open access is bad, but that peer review is a joke. If a nakedly bogus paper is able to get through journals that actually peer reviewed it, think about how many legitimate, but deeply flawed, papers must also get through. Any scientist can quickly point to dozens of papers – including, and perhaps especially, in high impact journals – that are deeply, deeply flawed – the arsenic DNA story is one of many recent examples. As you probably know there has been a lot of smoke lately about the "reproducibility" problem in biomedical science, in which people have found that a majority of published papers report facts that turn out not to be true. This all adds up to showing that peer review simply doesn't work.

While I agree that pre-publication peer review is inconsistent, I don't think it's broken. It does reasonably well when the reviewers and editors are on task and not just passing papers through a pipeline. Once in a while a paper like the above gets through a big name journal, where the checks on methods weren't strict enough, but for the most part, pre-publication peer review does exactly what it is supposed to.

My disagreement with all these folks is on the /purpose/ of pre-PR: it is NOT to judge the scientific value of a paper.

Pre-PR is, at best, a low pass filter. It insures that editors don't get shamed (often) for publishing gaffs, and that scientists in a particular field of research don't have to trudge through miles and miles of dreck just to find a paper that might be worthwhile. As such, reviewers and editors clean up the writing, make sure the methods follow from the introduction and the conclusion follows from all of the above, check logical consistency and reasoning, read the methods carefully, and look over the statistics. This is the job of pre-PR.

The job of judging the scientific /worth/ of a paper is what POST-publication peer review is for. I have spent many hours during my graduate career in these classes called "journal clubs". A group of students meet and one of us presents a paper, which we have all read, and the rest of the hour is spent tearing it to shreds. This is Post-PR, and it is absolutely necessary and far, far more important than Pre-PR could ever be. This is where the judgement comes in. All this training wasn't to make me a better pre-PR reviewer, it was to make me a post-PR reviewer who is not bound by the opinions of a few, usually anonymous people.

The problem is, people treat pre-PR as if it is the be all, end all of the peer review process, as if what is published is automatically worthy, or correct, or not fraudulent, just because it "passed peer review". The solution isn't to change the pre-PR process, the solution is to actually do post-PR review, to not simply trust the contents of a paper because it has Science or Nature or PLoSOne in the header. If I am simply taking what I find in an article as Word of God because some unnamed people got together and decided it was good enough to go, then what is /my/ worth as a scientist? How am I any more useful than people who take whatever media spin gets distributed?

This semester, I'm taking a journal club like class, with a bit more structure, but the main event every week is always a paper presentation and discussion. I spend HOURS carefully reading through these papers, even if I am not myself presenting. I go through the steps on this page, not only so I have good questions to ask, but so I train this skill until it becomes second nature. This is how every scientist should address a paper they are doing more than a skim over, or considering for use in their research, or as background for a paper, or if controversial. It is necessary, it is not just a game, and it plays a far more important role than the so called "broken" pre-publication peer review.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Nephew Twiddleton

Great post, Kai. I think that the idea that people seem to have that getting published somehow makes it a fact is part of the reason that people spout of shit about climate change and such. Though to be fair science reporting needs to improve as well.
Strange and Terrible Organ Laminator of Yesterday's Heavy Scene
Sentence or sentence fragment pending

Soy El Vaquero Peludo de Oro

TIM AM I, PRIMARY OF THE EXTRA-ATMOSPHERIC SIMIANS

Mesozoic Mister Nigel

It's really disheartening because the whole reason peer review exists is to prevent nonsense being published as science, and by falling down on the job journals are in effect not only failing to weed out the bullshit, but endorsing it.
"I'm guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk," Charles Wick said. "It was very complicated."


Kai

Quote from: Doktor Blight on October 13, 2013, 01:47:25 AM
Great post, Kai. I think that the idea that people seem to have that getting published somehow makes it a fact is part of the reason that people spout of shit about climate change and such. Though to be fair science reporting needs to improve as well.

Science reporting is darn good and gets better every year. The fine folks at Discover blogs and Nat Geo's Phenomenon do excellent work. The problem is, they are still overshadowed by the Old Media, like Huff Post, and do not get the exposure they deserve. I'm not quite sure why; maybe because hype rules media right now. Which is why I post stories by Ed Yong, Carl Zimmer, and Gwen Pearson on Facebook nearly ever day.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

LMNO


Kai

Quote from: LMNO, PhD (life continues) on October 13, 2013, 05:26:19 AM
I really like
The idea of
Propping up
A post-PR pR.

...I don't get it.

I know the abbreviations were less than optimal, but I think reading "pre-publication peer review" and "post-publication peer review" over and over would be worse.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

LMNO

Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

Kai

Quote from: LMNO, PhD (life continues) on October 14, 2013, 05:22:08 PM
Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

No problem.

I actually was inspired by my Bavaria-born professor (in all his spirited yet harmless contentiousness) to bring up my ideas in today's lab meeting. And of course they tore it to shreds: when they do pre-PR, they consider scientific worth, and so do editors. Except...when I told them that I don't trust a paper based on it's journal, nor do I believe it till I read it carefully, they said that's the right way to be.

Which seems contradictory. If post-PR is the way to do things, that suggests that pre-PR as a mediocre filter is fine. Not that I will perpetrate such a thing with the paper I am currently reviewing, nor with future papers. But it does suggest that the system isn't broken when it actually is present, and that what's broken is people's unwillingness to read critically, and only believe after careful consideration. I thought that growing up with Science had shaped my rational tendencies, but it seems once again that PD is responsible for good, or at least /intelligent/, things in my life.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Reginald Ret

Quote from: Kai on October 15, 2013, 12:16:39 AM
Quote from: LMNO, PhD (life continues) on October 14, 2013, 05:22:08 PM
Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

No problem.

I actually was inspired by my Bavaria-born professor (in all his spirited yet harmless contentiousness) to bring up my ideas in today's lab meeting. And of course they tore it to shreds: when they do pre-PR, they consider scientific worth, and so do editors. Except...when I told them that I don't trust a paper based on it's journal, nor do I believe it till I read it carefully, they said that's the right way to be.

Which seems contradictory. If post-PR is the way to do things, that suggests that pre-PR as a mediocre filter is fine. Not that I will perpetrate such a thing with the paper I am currently reviewing, nor with future papers. But it does suggest that the system isn't broken when it actually is present, and that what's broken is people's unwillingness to read critically, and only believe after careful consideration. I thought that growing up with Science had shaped my rational tendencies, but it seems once again that PD is responsible for good, or at least /intelligent/, things in my life.
You are the best combination of SCIENCE! and PD.
Lord Byron: "Those who will not reason, are bigots, those who cannot, are fools, and those who dare not, are slaves."

Nigel saying the wisest words ever uttered: "It's just a suffix."

"The worst forum ever" "The most mediocre forum on the internet" "The dumbest forum on the internet" "The most retarded forum on the internet" "The lamest forum on the internet" "The coolest forum on the internet"

Mesozoic Mister Nigel

Quote from: Kai on October 15, 2013, 12:16:39 AM
Quote from: LMNO, PhD (life continues) on October 14, 2013, 05:22:08 PM
Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

No problem.

I actually was inspired by my Bavaria-born professor (in all his spirited yet harmless contentiousness) to bring up my ideas in today's lab meeting. And of course they tore it to shreds: when they do pre-PR, they consider scientific worth, and so do editors. Except...when I told them that I don't trust a paper based on it's journal, nor do I believe it till I read it carefully, they said that's the right way to be.

Which seems contradictory. If post-PR is the way to do things, that suggests that pre-PR as a mediocre filter is fine. Not that I will perpetrate such a thing with the paper I am currently reviewing, nor with future papers. But it does suggest that the system isn't broken when it actually is present, and that what's broken is people's unwillingness to read critically, and only believe after careful consideration. I thought that growing up with Science had shaped my rational tendencies, but it seems once again that PD is responsible for good, or at least /intelligent/, things in my life.

I suspect that the reality is that both pre-and-post PR are necessary.
"I'm guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk," Charles Wick said. "It was very complicated."


Kai

Quote from: Not Your Nigel on October 16, 2013, 07:25:31 AM
Quote from: Kai on October 15, 2013, 12:16:39 AM
Quote from: LMNO, PhD (life continues) on October 14, 2013, 05:22:08 PM
Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

No problem.

I actually was inspired by my Bavaria-born professor (in all his spirited yet harmless contentiousness) to bring up my ideas in today's lab meeting. And of course they tore it to shreds: when they do pre-PR, they consider scientific worth, and so do editors. Except...when I told them that I don't trust a paper based on it's journal, nor do I believe it till I read it carefully, they said that's the right way to be.

Which seems contradictory. If post-PR is the way to do things, that suggests that pre-PR as a mediocre filter is fine. Not that I will perpetrate such a thing with the paper I am currently reviewing, nor with future papers. But it does suggest that the system isn't broken when it actually is present, and that what's broken is people's unwillingness to read critically, and only believe after careful consideration. I thought that growing up with Science had shaped my rational tendencies, but it seems once again that PD is responsible for good, or at least /intelligent/, things in my life.

I suspect that the reality is that both pre-and-post PR are necessary.

Necessary, yes. But I don't see them as having equal importance. Especially since for all but a few papers I have no control over pre-PR. But I can post-publication peer review any paper I want.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Mesozoic Mister Nigel

Quote from: Kai on October 16, 2013, 12:19:04 PM
Quote from: Not Your Nigel on October 16, 2013, 07:25:31 AM
Quote from: Kai on October 15, 2013, 12:16:39 AM
Quote from: LMNO, PhD (life continues) on October 14, 2013, 05:22:08 PM
Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

No problem.

I actually was inspired by my Bavaria-born professor (in all his spirited yet harmless contentiousness) to bring up my ideas in today's lab meeting. And of course they tore it to shreds: when they do pre-PR, they consider scientific worth, and so do editors. Except...when I told them that I don't trust a paper based on it's journal, nor do I believe it till I read it carefully, they said that's the right way to be.

Which seems contradictory. If post-PR is the way to do things, that suggests that pre-PR as a mediocre filter is fine. Not that I will perpetrate such a thing with the paper I am currently reviewing, nor with future papers. But it does suggest that the system isn't broken when it actually is present, and that what's broken is people's unwillingness to read critically, and only believe after careful consideration. I thought that growing up with Science had shaped my rational tendencies, but it seems once again that PD is responsible for good, or at least /intelligent/, things in my life.

I suspect that the reality is that both pre-and-post PR are necessary.

Necessary, yes. But I don't see them as having equal importance. Especially since for all but a few papers I have no control over pre-PR. But I can post-publication peer review any paper I want.

The advantage being that post-publication PR will make pre-publication PR tighter by necessity.
"I'm guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk," Charles Wick said. "It was very complicated."


Reginald Ret

Quote from: Not Your Nigel on October 17, 2013, 05:40:42 AM
Quote from: Kai on October 16, 2013, 12:19:04 PM
Quote from: Not Your Nigel on October 16, 2013, 07:25:31 AM
Quote from: Kai on October 15, 2013, 12:16:39 AM
Quote from: LMNO, PhD (life continues) on October 14, 2013, 05:22:08 PM
Hm.  October 12 was a Saturday.  The post was at 11:40 pm.


:checks: Yup, I was drinking. 


I think I was saying that doing additional peer review after a peer-reviewed paper is publish would be a good idea.  Which, I believe, is what is already happening.


So, yeah.  Feel free to ignore that one.

No problem.

I actually was inspired by my Bavaria-born professor (in all his spirited yet harmless contentiousness) to bring up my ideas in today's lab meeting. And of course they tore it to shreds: when they do pre-PR, they consider scientific worth, and so do editors. Except...when I told them that I don't trust a paper based on it's journal, nor do I believe it till I read it carefully, they said that's the right way to be.

Which seems contradictory. If post-PR is the way to do things, that suggests that pre-PR as a mediocre filter is fine. Not that I will perpetrate such a thing with the paper I am currently reviewing, nor with future papers. But it does suggest that the system isn't broken when it actually is present, and that what's broken is people's unwillingness to read critically, and only believe after careful consideration. I thought that growing up with Science had shaped my rational tendencies, but it seems once again that PD is responsible for good, or at least /intelligent/, things in my life.

I suspect that the reality is that both pre-and-post PR are necessary.

Necessary, yes. But I don't see them as having equal importance. Especially since for all but a few papers I have no control over pre-PR. But I can post-publication peer review any paper I want.

The advantage being that post-publication PR will make pre-publication PR tighter by necessity.
I would imagine you would want the Pre-publication PR to be looser, that way no good stuff gets filtered out.
Lord Byron: "Those who will not reason, are bigots, those who cannot, are fools, and those who dare not, are slaves."

Nigel saying the wisest words ever uttered: "It's just a suffix."

"The worst forum ever" "The most mediocre forum on the internet" "The dumbest forum on the internet" "The most retarded forum on the internet" "The lamest forum on the internet" "The coolest forum on the internet"

Kai

I'm not concerned about filtering out. With the explosion of open access journals, it's very unlikely that an article of worth will not find some publishing outlet. It's far more likely that an unworthy paper will find get published than the inverse. And like I said, I'm not concerned about that either. Rather, I'm concerned about the perception that peer review is infallible.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

LMNO

Quote from: Kai on October 17, 2013, 12:52:37 PM
I'm not concerned about filtering out. With the explosion of open access journals, it's very unlikely that an article of worth will not find some publishing outlet. It's far more likely that an unworthy paper will find get published than the inverse. And like I said, I'm not concerned about that either. Rather, I'm concerned about the perception that peer review is infallible.

Now that you mention it, there does seem to be an Appeal to Authority aura around peer review.  "The argument is valid.  It was peer reviewed!"