Author Topic: Failure to replicate earlier study causes original author to have a hissy fit.  (Read 3815 times)

Kai

  • A flea circus and
  • Deserved It
  • ****
  • Posts: 86960
  • Richard Feynman fangirl.
    • View Profile
Full summary over here: http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-study-doyen/

Basically, John Bargh, a researcher at Yale, conducted a psychological experiment to see if by talking about old age he could prime people to feel the effects of age (aka walking slower). This was back in 1996.

Just recently, Stephane Doyen attempted to replicate this study, but with blinding in some units, something that the original study did not have. Which ended up showing that the individuals walked slower "only when they were tested by experimenters who expected them to move slowly". A basic case of blinding revealing unconscious bias.

Bargh then wrote a blog post where he proceeded to throw a fit. Mind you, it's the sort of gentlemanly like fit that you see in academic circles, but most obviously a case of a grown man loosing it because he wasn't right. He attacks the journal, the authors, and Ed Yong as well. The last was a particularly dumb move, since Yong is probably the best damn science journalist out there, and wasn't exactly going to let it go at that.

This all illustrates that scientists aren't really different than other people when it comes to territoriality, and that replication, /published/ replication studies, are needed now more than ever. Until a study is replicated, it's a sample size one.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Nephew Twiddleton

  • Deserved It
  • ****
  • Posts: 259244
  • DEATH METAL ENGLISH
    • View Profile
Thats actually a really good idea. Any particular reason why follow up studies arent published?
Steely-Eyed Replicant Frottage Master of Yesterday's Lost Glory
Sentence or sentence fragment pending[/size]

Soy El Vaquero Peludo de Oro

TIM AM I, PRIMARY OF THE EXTRA-ATMOSPHERIC SIMIANS

Kai

  • A flea circus and
  • Deserved It
  • ****
  • Posts: 86960
  • Richard Feynman fangirl.
    • View Profile
Thats actually a really good idea. Any particular reason why follow up studies arent published?

Because there's this idiotic taboo against publishing negative results. The whole "you can't prove a negative" is deeply ingrained.

Quote from: The Art of Scientific Investigation - Beveridge pg 35
Commoner, however, is the failure of an experiment to demonstrate something because the exact conditions necessary
are not known, such as Faraday's early repeated failures to obtain an electric current by means of a magnet. Such experiments demonstrate the well-known difficulty of proving a negative proposition, and the folly of drawing definite conclusions from them is usually appreciated by scientists. It is said that some research institutes deliberately destroy records of " negative
experiments ", and it is a commendable custom usually not to publish investigations which merely fail to substantiate the hypothesis they were designed to test.

Now, okay, it's not all stupid. There are good reasons not to publish an experiment that failed to substantiate your personal hypothesis. But in the case of replicating someone else's work, a negation is even more important than an affirmation. Results that do not point to one hypothesis will point to another, even if that hypothesis is "our experiment was faulty".

But since it's already drilled into scientists (and therefore journal editors) not to publish negative results, they don't get published usually.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Nephew Twiddleton

  • Deserved It
  • ****
  • Posts: 259244
  • DEATH METAL ENGLISH
    • View Profile
Seems counterproductive if you cant publish follow up results that negate a previous hypothesis or at least throwit into question.
Steely-Eyed Replicant Frottage Master of Yesterday's Lost Glory
Sentence or sentence fragment pending[/size]

Soy El Vaquero Peludo de Oro

TIM AM I, PRIMARY OF THE EXTRA-ATMOSPHERIC SIMIANS

Mesozoic Mister Nigel

  • v=1/3πr2h
  • Deserved It
  • ****
  • Posts: 683676
  • The sky tastes like red exuberance.
    • View Profile
I understand the reasoning, I just don't agree with it... not in the age of internet publishing, where it need not be prohibitively expensive to publish studies that essentially just serve to say "Yo, previous study is not to be relied on".

In addition, I think there is an interesting double standard, in that studies that prove, for instance, that there is no link between Atrazine and cancer have no problem getting published.

And I also find the high drama in the scientific community hilarious.  :lulz:

I swear, I should start a journal called "The Journal of Negative Findings". It would be SO popular.
“I’m guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk,” Charles Wick said. “It was very complicated.”


Oysters Rockefeller

  • Semi-professional professional
  • Outlandish
  • ***
  • Posts: 5245
  • For no particular reason, beat up everyone.
    • View Profile
Seems counterproductive if you cant publish follow up results that negate a previous hypothesis or at least throwit into question.

Agreed.

"Yo, previous study is not to be relied on".

Man, it's not even like the reactor reached critical mass or some shit.
Well, my gynecologist committed suicide...
----------------------
I'm nothing if not kind of ridiculous and a little hard to take seriously.
----------------------
Moar liek Oysters Cockefeller, amirite?!

Mesozoic Mister Nigel

  • v=1/3πr2h
  • Deserved It
  • ****
  • Posts: 683676
  • The sky tastes like red exuberance.
    • View Profile
Also, I realize that behavioral sciences are a bit different, but I really took issue with the "There are many reasons for an experiment not to work" because findings that don't confirm your hypothesis does not equal "not working". How many fucking times did Bacall and Davis re-run their numbers and very expensive experiment? It was working FINE, it just didn't yield the results they were expecting, which led to an even more significant discovery.
“I’m guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk,” Charles Wick said. “It was very complicated.”


Q. G. Pennyworth

  • Slimy Thing Who
  • Deserved It
  • ****
  • Posts: 79633
  • QUEEN BITCH OF FLYERS
    • View Profile
I swear, I should start a journal called "The Journal of Negative Findings". It would be SO popular.

This should be a thing. Let's find some scientists and make it a thing.
Overheating Pheremone Pustule of Last Saturday's Jiggle Fun| _xgeWireToEvent: Unknown extension 131, this should never happen.

Don't fucking judge me, I've got tentacles for a face.

Mesozoic Mister Nigel

  • v=1/3πr2h
  • Deserved It
  • ****
  • Posts: 683676
  • The sky tastes like red exuberance.
    • View Profile
I swear, I should start a journal called "The Journal of Negative Findings". It would be SO popular.

This should be a thing. Let's find some scientists and make it a thing.

I'm gonna try to do it. Seriously. If I don't forget.
“I’m guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk,” Charles Wick said. “It was very complicated.”


Kai

  • A flea circus and
  • Deserved It
  • ****
  • Posts: 86960
  • Richard Feynman fangirl.
    • View Profile
I understand the reasoning, I just don't agree with it... not in the age of internet publishing, where it need not be prohibitively expensive to publish studies that essentially just serve to say "Yo, previous study is not to be relied on".

In addition, I think there is an interesting double standard, in that studies that prove, for instance, that there is no link between Atrazine and cancer have no problem getting published.

And I also find the high drama in the scientific community hilarious.  :lulz:

I swear, I should start a journal called "The Journal of Negative Findings". It would be SO popular.

Well, there's already a Journal of Irreproducible Results, so it should go over well.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Mesozoic Mister Nigel

  • v=1/3πr2h
  • Deserved It
  • ****
  • Posts: 683676
  • The sky tastes like red exuberance.
    • View Profile
I understand the reasoning, I just don't agree with it... not in the age of internet publishing, where it need not be prohibitively expensive to publish studies that essentially just serve to say "Yo, previous study is not to be relied on".

In addition, I think there is an interesting double standard, in that studies that prove, for instance, that there is no link between Atrazine and cancer have no problem getting published.

And I also find the high drama in the scientific community hilarious.  :lulz:

I swear, I should start a journal called "The Journal of Negative Findings". It would be SO popular.

Well, there's already a Journal of Irreproducible Results, so it should go over well.

The JIR is always entertaining. Nerdery in general is always entertaining, especially when nerds quibble.

This morning I was reading what may have been the pettiest debate ever, about the best longhand method of finding a square root. There was name-calling. It was amazing. I think that a journal that published nothing but negative findings would become a major locus for science drama.

“I’m guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk,” Charles Wick said. “It was very complicated.”


Mesozoic Mister Nigel

  • v=1/3πr2h
  • Deserved It
  • ****
  • Posts: 683676
  • The sky tastes like red exuberance.
    • View Profile
I also really enjoy the hell out of the Annals of Improbable Research, which most people are probably familiar with thanks to the Ig-Nobels and the Luxuriant Flowing Hair Club for Scientists (which I am looking forward to joining, once I am a scientist).
“I’m guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk,” Charles Wick said. “It was very complicated.”


Kai

  • A flea circus and
  • Deserved It
  • ****
  • Posts: 86960
  • Richard Feynman fangirl.
    • View Profile
I also really enjoy the hell out of the Annals of Improbable Research, which most people are probably familiar with thanks to the Ig-Nobels and the Luxuriant Flowing Hair Club for Scientists (which I am looking forward to joining, once I am a scientist).

I have yet to join the LFHCS. This is a moral imperative.
If there is magic on this planet, it is contained in water. --Loren Eisley, The Immense Journey

Her Royal Majesty's Chief of Insect Genitalia Dissection
Grand Visser of the Six Legged Class
Chanticleer of the Holometabola Clade Church, Diptera Parish

Mesozoic Mister Nigel

  • v=1/3πr2h
  • Deserved It
  • ****
  • Posts: 683676
  • The sky tastes like red exuberance.
    • View Profile
I also really enjoy the hell out of the Annals of Improbable Research, which most people are probably familiar with thanks to the Ig-Nobels and the Luxuriant Flowing Hair Club for Scientists (which I am looking forward to joining, once I am a scientist).

I have yet to join the LFHCS. This is a moral imperative.

I can't even believe that you're not already a member! I kind of just assumed that you were.
“I’m guessing it was January 2007, a meeting in Bethesda, we got a bag of bees and just started smashing them on the desk,” Charles Wick said. “It was very complicated.”


hirley0

  • Deserved It
  • ****
  • Posts: 36261
    • View Profile
    • TNM
 :fnord: ~o~ v read top down v
SO: 2 sepant / 21March N. i'poise the prime problem R thumbnails
7:47Ab while its true i do NOT know what to do HERE is A link Anyway

:fnord: 3/21 ? thumbnail photo ?

http://sf0.org/SFmedia/badges/newplayer.gif & 18
  {18X18 piX
 
http://media.turnofspeed.com/media/hub/square_26101659.jpg
{60x60

http://media.turnofspeed.com/media/hub/main_26101659.jpg
{400x320

640x480
http://photos3.meetupstatic.com/photos/member/4/f/f/d/highres_12620477.jpeg
« Last Edit: March 21, 2012, 04:50:44 pm by hirley0 »