Failure to replicate earlier study causes original author to have a hissy fit.

[Cain]

Last month, I learned about a publication that has been quickly gaining popularity, the Journal of Negative Results in BioMedicine (JNRBM). Published, presumably, by a gang of dour curmudgeons who hate everything, JNRBM openly welcomes the data that other journals won't touch because it doesn't fit the unspoken rule that all articles must end on a cheery note of promise. ("This could lead to new therapies!" boast most journal articles, relying on the word "could" to keep their platitudes accurate and the exclamation point to boost excitement, stand for "factorial," or make a clicking sound, depending on your field.)

You might imagine that JNRBM is a place where losers gather to celebrate their failures, kind of like Best Buy or Division III football. But JNRBM meets two important needs in science reporting: the need to combat the positive spin known as publication bias and the need to make other scientists feel better about themselves.

(Unfortunately, if you don't work in biomedicine, you're still screwed. The Journal of Negative Results in Zoology, for example, is just called "not seeing animals." And the Journal of Negative Results in Homeopathy is the entire field of homeopathy.)

When it comes time to put our science into words, why do we pretend that the negative results never happened? Why do we have so much trouble accepting that sometimes our hypotheses are disproved? But most importantly, where was this freaking journal when I was in grad school? You can get published even when the experiment fails—it's the easiest way to pad your CV since the invention of 1.25-inch margins.

http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2012_02_24/caredit.a1200021

[Cain]

The Journal for Negative Results in Political Science is also referred to as "history", for those wondering.

[Mesozoic Mister Nigel]

The Journal for Negative Results in Political Science is also referred to as "history", for those wondering.

[Mesozoic Mister Nigel]

Last month, I learned about a publication that has been quickly gaining popularity, the Journal of Negative Results in BioMedicine (JNRBM). Published, presumably, by a gang of dour curmudgeons who hate everything, JNRBM openly welcomes the data that other journals won't touch because it doesn't fit the unspoken rule that all articles must end on a cheery note of promise. ("This could lead to new therapies!" boast most journal articles, relying on the word "could" to keep their platitudes accurate and the exclamation point to boost excitement, stand for "factorial," or make a clicking sound, depending on your field.)

You might imagine that JNRBM is a place where losers gather to celebrate their failures, kind of like Best Buy or Division III football. But JNRBM meets two important needs in science reporting: the need to combat the positive spin known as publication bias and the need to make other scientists feel better about themselves.

(Unfortunately, if you don't work in biomedicine, you're still screwed. The Journal of Negative Results in Zoology, for example, is just called "not seeing animals." And the Journal of Negative Results in Homeopathy is the entire field of homeopathy.)

When it comes time to put our science into words, why do we pretend that the negative results never happened? Why do we have so much trouble accepting that sometimes our hypotheses are disproved? But most importantly, where was this freaking journal when I was in grad school? You can get published even when the experiment fails—it's the easiest way to pad your CV since the invention of 1.25-inch margins.

http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2012_02_24/caredit.a1200021

This is fucking AWESOME.     

[hirley0]

Read 253 times 5:15 pdT
Warning - while you were typing 2 new replies have been posted. You may wish to review your post.

v read top down v

  {18X18 piX

[Mesozoic Mister Nigel]

I am, however, kind of bummed that someone has already started that journal, and it's even in the field I want to go into.

[hirley0]

Four OFFered

I am, however, kind of bummed that someone has already started that journal, and it's even in the field I want to go into.

[Cain]

I am, however, kind of bummed that someone has already started that journal, and it's even in the field I want to go into.

On the other hand, you can start submitting papers already.  Get falsifying!

[hirley0]

? TWO ? Read 262 times)

[Mesozoic Mister Nigel]

I am, however, kind of bummed that someone has already started that journal, and it's even in the field I want to go into.

On the other hand, you can start submitting papers already.  Get falsifying!

[Rococo Modem Basilisk]

The Language Log posted a very good article on the subject of the non-replication mentioned in the OP, and what it does and does not mean. It does not involve itself with Bargh's rebuttals (since that's entirely unrelated to the point -- Bargh's study was considered a landmark study and strongly influenced much of cogsci, in addition to being repeatedly cited both in academic works in the field and in works for a general audience). The original is here, but because it's such a good article (and because many of you are likely to be too lazy to click the link, and will continue to argue about the tangential issue of whether or not the first guy on the authors list of the original study is an asshole) I will quote it in its entirety here.

Replication Rumble
March 17, 2012 @ 3:16 pm · Filed by Julie Sedivy under Psychology of language

In other non-replication news lately: There's been a pretty kerfuffle this month in social psychology and science blogging corners over a recent failure to replicate a classic 1996 study of automatic priming by John Bargh, Mark Chen, and Lara Burrows. The non-replication drew the attention of science writer Ed Yong who blogged about it over at Discover, and naturally, of John Bargh, who elected to write a detailed and distinctly piqued rebuttal at Psychology Today.

The original paper reported three experiments; the one that's the target of controversy used a task in which subjects unscramble lists of words and isolate one word in the list that doesn't fit into the resulting sentence. The Bargh et al. study showed that when the experimental materials contained words that were associated with stereotypes of the elderly (e.g. Florida, bingo, gray, cautious), subjects walked more slowly down the hall upon leaving the lab compared to subjects who saw only neutral words. The result has been energetically cited, and has played no small role in spawning a swarm of experiments documenting various ways in which behavior can be impacted by situational or subliminal primes. The authors explained their findings by suggesting that when the concept of a social stereotype is activated (e.g. via word primes), this can prompt behaviors that are associated with that stereotype (e.g. slow walking).

But allegedly, despite scads of studies that have built on some of Bargh et al.'s conclusions, the slow-walker study has yet to be fully replicated, which motivated Stéphane Doyen and colleagues at the Université Libre de Bruxelles to undertake the job, reporting their attempts in a recently-published article in PLoS ONE. Their first experiment, which contained sober experimental precautions such as using automated timing systems and ensuring that experimenters were blind as to which conditions subjects were assigned to, failed to produce a priming effect. This led them to wonder in print whether the original priming results could have come from a failure to strictly implement double-blind experimental methods, which serves as the motivation for their second experiment.

The second study reported by Doyen et al. focused on whether an effect could be induced by specifically manipulating the experimenters' expectations of how the subjects would behave. Ten different experimenters were included; these experimenters were made aware of which of their subjects were assigned to the word prime condition, and which were assigned to the neutral word condition. However, half of them were led to expect that when primed, their subjects would walk more slowly as a result of the experimental manipulation, and half of them were led to believe they would walk more quickly. (In reality, all subjects received the same elderly-priming materials as in the first experiment). The paper doesn't go into detail as to how these experimenter expectations were established, other than to report that all this took place during "a one hour briefing and persuasion session prior to the first participant's session." In addition, the experimenters had their expectations reinforced by the behavior of their very first study subject, who was a confederate in cahoots with the researchers and obligingly walked quickly or slowly, as expected.

Not surprisingly, when subjects' walking speed was measured by the experimenters themselves on a stopwatch, their pace aligned with expectations: subjects in the word prime condition were timed at faster speeds than those in the neutral word condition when the experimenters expected that priming would speed them up, and conversely, when experimenters expected priming to slow the subjects down, they timed them at slower speeds in the word prime condition relative to the neutral word condition. This wasn't the whole story, though—the subjects' actual speed was also timed by an automated motion-sensitive system. Objective measures of walking speed showed that when the experimenters expected priming to accelerate their subjects, that's exactly what happened. But when they expected subjects to slow down as a result of the priming, there was no difference between the primed subjects and those in the neutral word condition.

This tells us that the actual walking speed of subjects isn't determined entirely by experimenters' expectations; if that were the case, subjects should have walked more quickly when expected to do so as a result of priming. But it does suggest that the priming effect can be either boosted or dampened by experimenter expectations-presumably because the experimenter is emitting subtle and possibly inadvertent cues that impact the subjects' behavior (it would have been interesting, for example, to measure the experimenters' speech rate).

The authors' take on all this is to conclude that:

although automatic behavioral priming seems well established in the social cognition literature, it seems important to consider its limitations. In line with our result it seems that these methods need to be taken as an object of research per se before using it can be considered as an established phenomenon.

I'm really not sure what the above statement actually means. But it certainly invites a first-blush response of the Ohmygosh-is-all-this-stuff-we-thought-we-knew-about-unconscious behavioral-priming-wrong? variety. But it's worth waiting for that first flush to settle. Because in the end, the result in and of itself causes little trauma to the original Bargh et al. interpretation of their priming data, and none whatsoever to the more general issue of whether automatic behavioral priming exists.

First of all, the fact that experimenter expectations led to an effect on subjects' behavior doesn't mean that this accounts for the original Bargh et al. results. It just means that it has a measurable impact on any priming effects that may or may not occur. To find otherwise would be rather surprising, especially given the rather heavy-handed way in which these expectations seem to have been induced. (Bargh has countered the paper by claiming that in fact, their own study did implement double-blind methods; whether or not this was done rigorously enough, it certainly seems clear that the later Doyen et al. paper went to special lengths to create a salient experimenter bias above and beyond what would plausibly have existed in the earlier work).

So what we're really left with is the issue of how to interpret the non-replication. There are a number of possible reasons for this, some of them really boring, some of them mildly interesting, but most of them unrelated to the important theoretical questions. For example:

1. The non-replication itself is an experimental failure. In experiments involving humans, all "replications" are at best approximate. Other unforeseen aspects of the experimental design and implementation may have obscured a priming effect or led to unusually noisy data. For example, maybe the Belgian experimenter was attractive to the point of distraction. Maybe more of the undergraduate subjects were tested in the morning while still sluggish. Maybe the experimenter was flaky and inconsistent in implementing the study. Obviously, if an effect is repeatedly vulnerable to these kinds of obliterations, that can speak to the fragility of the effect; but the point is that for any single failure to replicate, we can't tell for sure what the source of the non-replication is. Perfectly robust results can be and often are drowned in noise inadvertently introduced somewhere in the experimental procedure. We can simply document that the failure to replicate occurred, while noting (and further testing) any obvious discrepancies from the original implementation.

2. The word primes may not have successfully triggered a stereotype for the elderly in the minds of the subjects, or the conceptual stereotype may not have had a strong association with slow walking movements. It's entirely conceivable that stereotypes would shift due to time or geographic location. A lot has happened demographically since 1991 when Bargh et al. first collected their data. Upon hearing about this study, for example, my own son remarked (referring to his alpine-skiing, Nepal-trekking grandmother): "Those subjects have obviously never met Nanny." In this case, there's no threat to Bargh's original theoretical contribution about the activation of social stereotypes as a driver of behavior; it's just that any given stereotype isn't going to be held by all populations.

3. There was nothing wrong with the stereotypes; the original result really was a statistical fluke, or an experimental artifact, or limited to a very narrow population or set of experimental circumstances. This eventuality is the most damaging to Bargh et al. But does it really threaten the more general conclusion that behavior can be unconsciously, or automatically primed? No; it simply casts doubt on the more specific interpretation of the results as being due to the activation of social stereotypes. In fact, it's hard to interpret Doyen et al.'s second study, which manipulated experimenter expectations, without appealing to unconscious behavioral priming (as fairly pointed out by Ed Yong in his post). Unless the experimenters actually violated experimental ethics outright by instructing the subjects to walk more slowly, it seems likely that the subjects were unconsciously picking up on experimenter cues (but which ones? Speech rate? Certain words?) unconsciously emitted by the experimenters. What's more, there are by now dozens and quite possibly hundreds of demonstrations of automatic priming effects using a variety of different experimental paradigms, some of which do apply the activation of stereotypes. (Some examples here and here.) Given that it's now 2012, not 1996 when the Bargh et al. paper first appeared, any non-replication of that original result is going to have to be interpreted within the context of that entire body of work.

So. Hardly material to launch a full-scale kerfuffle. This is just science plodding its plodding way towards its plodding approximation of truth. Enough with the rubbernecking already—there are no bloody conclusions to be found here, at least not yet.

So why am I bothering to add my voice to the fray? Because I think that it's very important that we actually talk about replication, what it means and doesn't mean, and that we do so in a way that moves beyond thinking about it as a cagematch between scientists.

When I talk to non-scientists, I'm distressed by a general illiteracy in the understanding of non-replication. All too often, failures to replicate are treated as abrupt reversals of truth. As if any new result, especially a startling or counterintuitive one, were anything but an opening gambit, not a declaration of truth. New studies, whether they replicate the result or not, are simply the next moves that change the way the board is now configured. But all too often, a failure to replicate is portrayed as an instance of science "changing its mind" or an indictment of the scientific method, when really, it's at the heart of the scientific method. When it comes down to it, the sound of non-replication isn't the sound of the puck being slapped into the opponent's net. It's the sound of a muttered "hmmm, what's going on here," the sound of science rolling up its sleeves with a sigh and settling in for a long night's work.

March 17, 2012 @ 3:16 pm · Filed by Julie Sedivy under Psychology of language

The TL;DR version: the original study was attempting to analyze the effect of cultural stereotypes in primed behaviors. A later study performed in a completely different country using a completely different language failed to get similar results, which led to an analysis of other variables. The result is that it seems that in Belgium, French-speaking students are more strongly influenced in their behavior by the expectations of the experimenters than by their stereotype of old people, whereas in an earlier study at Yale the expectations of the experimenters were not adequately controlled for and may have influenced results they thought could only be attributed to the stereotypes Yale psychology students had about old people.

This is arguably a good thing, because it will lead to exploration of unconscious use of non-verbal communication and its effects on primed responses, and because it will bring life back into the study of the effects of social stereotypes on primed responses (and probably how this differs between cultures). However, some media outlets and other dumb people will spin this as refutation (or even, if they are dumb enough, disproof) of primed responses.

[Kai]

I'm far more concerned with the lack of replication and the distaste for replication with negative results

than I am for people who grasp every negative result as a disproof. Or more likely, latch onto results when they follow biases and discard when they don't.

The latter, well, people will be primates.

The former is something I can work to change.

[hirley0]

GOOD D-