News:

Endorsement:  I am not convinced you even understand my concepts of moral relativity, so perhaps it would be best for you not to approach them.

Main Menu

Journalist submits fake paper, passes peer review.

Started by Kai, October 13, 2013, 01:05:29 AM

Previous topic - Next topic

Mesozoic Mister Nigel

Quote from: Kai on October 17, 2013, 12:52:37 PM
I'm not concerned about filtering out. With the explosion of open access journals, it's very unlikely that an article of worth will not find some publishing outlet. It's far more likely that an unworthy paper will find get published than the inverse. And like I said, I'm not concerned about that either. Rather, I'm concerned about the perception that peer review is infallible.

What Kai said.
"I'm guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk," Charles Wick said. "It was very complicated."


Mesozoic Mister Nigel

Quote from: LMNO, PhD (life continues) on October 17, 2013, 03:48:19 PM
Quote from: Kai on October 17, 2013, 12:52:37 PM
I'm not concerned about filtering out. With the explosion of open access journals, it's very unlikely that an article of worth will not find some publishing outlet. It's far more likely that an unworthy paper will find get published than the inverse. And like I said, I'm not concerned about that either. Rather, I'm concerned about the perception that peer review is infallible.

Now that you mention it, there does seem to be an Appeal to Authority aura around peer review.  "The argument is valid.  It was peer reviewed!"

yes, and the concept behind it actually should make that appeal sound; essentially, a non-expert should be able to point to a peer-reviewed article and say "I am not an expert, but a panel of experts found this research methodologically sound so I am offering it to support my position on X". Unfortunately, for whatever reason, a lot of peer reviewers seem to be falling down on the job, and in some cases don't even seem to be reading the articles they're passing. This is a huge problem, not just for the layperson but for other scientists who are basing their research on the existing body of research as communicated in, you guessed it, peer-reviewed papers.
"I'm guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk," Charles Wick said. "It was very complicated."


The Good Reverend Roger

Quote from: Not Your Nigel on October 17, 2013, 04:41:43 PM
Quote from: LMNO, PhD (life continues) on October 17, 2013, 03:48:19 PM
Quote from: Kai on October 17, 2013, 12:52:37 PM
I'm not concerned about filtering out. With the explosion of open access journals, it's very unlikely that an article of worth will not find some publishing outlet. It's far more likely that an unworthy paper will find get published than the inverse. And like I said, I'm not concerned about that either. Rather, I'm concerned about the perception that peer review is infallible.

Now that you mention it, there does seem to be an Appeal to Authority aura around peer review.  "The argument is valid.  It was peer reviewed!"

yes, and the concept behind it actually should make that appeal sound; essentially, a non-expert should be able to point to a peer-reviewed article and say "I am not an expert, but a panel of experts found this research methodologically sound so I am offering it to support my position on X". Unfortunately, for whatever reason, a lot of peer reviewers seem to be falling down on the job, and in some cases don't even seem to be reading the articles they're passing. This is a huge problem, not just for the layperson but for other scientists who are basing their research on the existing body of research as communicated in, you guessed it, peer-reviewed papers.

Yep.  So now I basically can't even trust the scientific community.  May as well listen to advertizing execs and religious whackjobs...Because it's the same fucking thing, when you get down to brass tacks.

You're hearing what someone WANTS you to believe is the truth, not the truth.

It's fucking disgusting.
" It's just that Depeche Mode were a bunch of optimistic loveburgers."
- TGRR, shaming himself forever, 7/8/2017

"Billy, when I say that ethics is our number one priority and safety is also our number one priority, you should take that to mean exactly what I said. Also quality. That's our number one priority as well. Don't look at me that way, you're in the corporate world now and this is how it works."
- TGRR, raising the bar at work.

The Good Reverend Roger

Quote from: LMNO, PhD (life continues) on October 17, 2013, 03:48:19 PM
Quote from: Kai on October 17, 2013, 12:52:37 PM
I'm not concerned about filtering out. With the explosion of open access journals, it's very unlikely that an article of worth will not find some publishing outlet. It's far more likely that an unworthy paper will find get published than the inverse. And like I said, I'm not concerned about that either. Rather, I'm concerned about the perception that peer review is infallible.

Now that you mention it, there does seem to be an Appeal to Authority aura around peer review.  "The argument is valid.  It was peer reviewed!"

AH, MY PEOPLE, I LOVE YOU!
\
" It's just that Depeche Mode were a bunch of optimistic loveburgers."
- TGRR, shaming himself forever, 7/8/2017

"Billy, when I say that ethics is our number one priority and safety is also our number one priority, you should take that to mean exactly what I said. Also quality. That's our number one priority as well. Don't look at me that way, you're in the corporate world now and this is how it works."
- TGRR, raising the bar at work.

Kai

Okay. I'm done with the butthurt. Here's some food for Germans.

http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

QuoteAcademic scientists readily acknowledge that they often get things wrong. But they also hold fast to the idea that these errors get corrected over time as other scientists try to take the work further. Evidence that many more dodgy results are published than are subsequently corrected or withdrawn calls that much-vaunted capacity for self-correction into question. There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think.

Various factors contribute to the problem. Statistical mistakes are widespread. The peer reviewers who evaluate papers before journals commit to publishing them are much worse at spotting mistakes than they or others appreciate. Professional pressure, competition and ambition push scientists to publish more quickly than would be wise. A career structure which lays great stress on publishing copious papers exacerbates all these problems. "There is no cost to getting things wrong," says Brian Nosek, a psychologist at the University of Virginia who has taken an interest in his discipline's persistent errors. "The cost is not getting them published."

The whole article is on mistakes and falsehoods in scientific publishing, and why replications (which are a kind of post publication peer review) are absolutely necessary and not happening. And you know what? This DOES upset me. I accept completely that peer reviewed journals are going to slip up sometimes, that peer reviewers are going to fail, that mistakes and falsehoods are going to be published. It happens, it's going to continue to happen, there's not a damn thing anyone can do to eliminate it completely. Which is why follow ups are so damn important.

Maybe Science really /is/ broken/short circuit, and if it IS, then the broken part is that it's become like media. The entire point is to pour out stories, with not a bit of thought to questioning whether the stories that just got poured out were any good. THATS the supposed self correcting, and since we've been letting the journalists do it FOR us, the letters are still PR but pronounced "public relations" and not "peer review". This is disturbing. And I don't know fuck all I can do about it.

Also, I've been wondering who the hell that guy in the picture is.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

The Good Reverend Roger

" It's just that Depeche Mode were a bunch of optimistic loveburgers."
- TGRR, shaming himself forever, 7/8/2017

"Billy, when I say that ethics is our number one priority and safety is also our number one priority, you should take that to mean exactly what I said. Also quality. That's our number one priority as well. Don't look at me that way, you're in the corporate world now and this is how it works."
- TGRR, raising the bar at work.

Kai

Quote from: Dirty Old Uncle Roger on October 24, 2013, 06:34:58 PM
Andrew Wakefield.

Oh. Well, what's worse, finding out about him later, or never finding out? Even if peer review works 99.9% of the time, 0.1% are still going to get through. Admittedly, the numbers are worse than that. The difference between the two is that in one, Science is aware of this and works to self correct it, and in the other, Science ignores the ever constant dilemma of peer review. I refuse to pretend that it can be perfect, that would really make the assholes "my people".
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Kai

I just realized I need some help with this figure. The proportions don't seem right. Need some Bayes up in this here thing. LMNO?

If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

LMNO

At first blush, something does seems off.  I can't see exactly what it is, however.

Kai

It's the false positives. 5% rate of false positives is 5% of those studies that had significant results, not five percent of the total, right?
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

LMNO


LMNO

Wait. 100 "true things", 5% error rate...


Ah! Where's the rigor? Shouldn't we be testing more than once, if we have a known error rate?

Stupid Youngin

#27

The Good Reverend Roger

That is an EXCELLENT work, and the sources provided are pretty Goddamn good.
" It's just that Depeche Mode were a bunch of optimistic loveburgers."
- TGRR, shaming himself forever, 7/8/2017

"Billy, when I say that ethics is our number one priority and safety is also our number one priority, you should take that to mean exactly what I said. Also quality. That's our number one priority as well. Don't look at me that way, you're in the corporate world now and this is how it works."
- TGRR, raising the bar at work.

Kai

Quote from: LMNO, PhD (life continues) on October 24, 2013, 10:28:05 PM
Wait. 100 "true things", 5% error rate...


Ah! Where's the rigor? Shouldn't we be testing more than once, if we have a known error rate?

Something is just not right about the middle part of that figure. It needs some Bayes-jutsu.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish