Who wants to play Evolutionary Neuro Cognitive Research FAIL?

Let’s all imagine that we’re cognitive neuroscientists and we want to do some “research” about fanfic (why fanfic? nobody knows!) and see if we can get a bunch of womengurlz to support our pet theories about “the unified fabric of human desire” (whatever that is — ilithiana says plaid). Because you can totally tell stuff about brain function from hacked-together surveys on Appspot.

What will we put on our survey? Here are my questions.

1. What sex are you?
a) Man. 100% manly man. GRRR.  
b) Female. *teehee*  
c) Confused.

2. Which statement do you agree with? Choose one:
a) I love cock!
b) All men are heterosexual.
c) One day my prince will come, and he will be Edward Cullen.

3. Which best represents your fanfic reading habits?
a) I fulfil my personal fantasies by inserting myself into fictional scenarios.
b) Because of my sexual inexperience, I read fanfic as research about boys.
c) I read fanfic because I am into depraved kinks like homosexuality and bandom.

Jonquil (who, incidentally, is kicking bottoms and taking names on this one — check recent entries on her journal) suggested via IM:

4. Will you please tell me about your sexual practices? With pictures?

If you need inspiration, check out this transcription of the 70-question survey. Remember, nothing you suggest will ever be reviewed by an IRB, so you can ask anything.

See also: Ten steps to a perfect fanstorm at Hoyden About Town, unfunnybusiness roundup, linkspam roundup on DW, high-larious Ogi/Sai badfic slash (NSFW).

Photo credit: innocentsmith @ dreamwidth

Credit: innocentsmith @ dreamwidth

In conclusion: fandom, I love you. You are smart and funny and don’t take shit from anyone — especially not cave-dwelling neanderthals posing as scientists.

18 thoughts on “Who wants to play Evolutionary Neuro Cognitive Research FAIL?

  1. Mel

    Thank you for this post together!

    I was quite overwhelmed by the amount of links and how far this fail has flung.

  2. Leigh Honeywell

    And now the LJ account the researchers were using is friends-only. Somehow, that they did that rather than just deleting is, is super duper ironic to me.

  3. Bene

    I’m just stunned at how deep of a hole these guys dug and kept digging for themselves. The mind, she boggles.

  4. Asad

    I’ve spent the last couple of evenings reading all the links on this with considerable fascination. Even as an outsider to fanfic/online fandom, I agree that what Ogas and Gaddam at best made a very poor impression. However, I looked at some of the critiques with mixed feelings, especially from the critics who came from an academic standpoint:

    1. Part of the firestorm clearly became another substitute for the EvPsych Warz. While I myself have come to disagree with the naïve adaptationism that seems to pervade EvPsych thinking and find their specific project about gender/sex in particular questionable*, I think that the search for cognitive universals underlying human behaviour is a useful and valid one.

    I understand that some find the very concept of universals in social cognition to be threatening to their political projects, but that doesn’t mean that they should not be studied. I got the impression that eg Eruthros in particular, and certainly the critics coming from a social science/humanities perspective seemed to think they shouldn’t, in principle. But there has to be an underlying model for us to function as social beings at all.

    2. It turned out to be the case that what Ogas and Gaddam were actually attempting was some kind of cognitive modeling via machine learning as a substitute for neuroimaging. Some of the critics who appeared to have cognitive science backgrounds seemed to reject this methodology because they felt that social cognition is too subtle to be captured by surveys. While after reading the survey, I suspect it was too coarse a tool (…), it’s not necessarily the case that no survey could have done it—and in fact that’s what Ogas and Gaddam had set out to do.

    3. While it turns out that Ogas and Gaddam are no longer associated with BU and the IRB has nothing to do with it, I can kind of easily imagine what went through their heads when they were confronted with this criticism… Machine learning/natural language processing is a grey area when it comes to IRBs and data collection. There is some amount of resistance to viewing machine learning experiments based on data gathered from human providers to be “human subjects research” and subject to the same guidelines. Though their survey itself put them well out of that grey area, notwithstanding sophistry.

    In sum, I thought there were two “tiers” of criticism, one about their behaviour and attitudes towards the community with which they were interacting, and one about the experiment itself, and the latter reaction I’m more dubious about than the former.

    *Some time ago I thought it made perfect sense…in high school after having been exposed to the “Family Values” chapter in Pinker’s How the Mind Works. It took me some time to realize the flaws in the argument, and things like the Berwick and Ahouse review of his book…

    1. Skud Post author

      Asad: I don’t disagree with you that there is an undercurrent of “this shouldn’t be studied”, but I see it in a larger context of, “… because whenever this is studied, the results are used against women.” To me, it’s like someone coming in and saying “We’re studying eugenics!” It’s going to have some really nasty connotations and unless the researcher’s gone to some really SERIOUS effort to mitigate that, I would have no particular reason to think that they weren’t going to mis-use the results.

      1. Anneke

        To me, it’s like someone coming in and saying “We’re studying eugenics!”

        OMG YES THIS.

        I don’t know if you’re familiar with Joanna Russ’s book To Write Like a Woman, but there were bits and pieces of it flashing through my brain as I read all this.

  5. Meg Thornton

    Meddle not in the affairs of fandom, for we are not particularly subtle, are definitely quick to anger, and you look good on the bottom. Or as the “meat” in a sandwich. Or riding the train. Or however else you’re pictured in the inevitable badfic which will go along with all other examples of “point and mock”.

    Oh, and in answer to one of their survey “questions” – my sexual orientation is either horizontal, or west. Depending on whether they’re measuring an absolute or a relative orientation.

  6. Asad

    Here’s a really good explanation on LJ that pretty much matches with my impression of what they were trying to do. From their point of view, they were only looking to elicit data from a particular community, but it didn’t matter so much what that data actually was. They alluded repeatedly to plans to elicit the same data from a community they believe (and are probably correct) to be very different from the fanfic community.

    They will probably construct and train two models based on the two sets of elicited data, and their hypothesis is that the models will “react” in a certain way to the stimuli they plan to subject the models to. They changed questions simply as the need for training data changed—it was not intended for conventional survey purposes, so the “scientificness” of this doesn’t matter to them (they think). Fake answers provide data too, as long as they were “genuine” fake answers that the other population they plan to use for data provides different data. In essence, it really was a “they meant to do that” research troll from the outset.

    I part from neededalj in that I don’t view it with as much incredulity as s/he does. Oh, I don’t doubt that the concept is flawed and premature, but the idea of a cognitive modeling project via machine learning over elicited text does not yield a lot of surprise from my corner at least. As I suggested above, one thing I found very interesting was that the incredulity came from social scientists and neuroscientists and like-minded people.

    I don’t doubt the concept is deeply flawed, as I said, but it’s neither as crazy nor as stupid as some of their critics are making it to be. This doesn’t excuse their behaviour either.

    1. Skud Post author

      They alluded repeatedly to plans to elicit the same data from a community they believe (and are probably correct) to be very different from the fanfic community.

      They can’t elicit the same data from another community, because many of their questions were specific to fanfic. Of what other community could they ask, “At what age did you read your first slash story?” or “Which specific types of fan fiction do you actively seek out to read?”

      1. Asad

        Oh, my point was that they don’t need the *same* data. They just need textual data that’s functionally equivalent for their purposes. They’re training a model, not analyzing the results of a survey.

        Of course, the flaw lies in their notion of functional equivalence which has been teased out at length in those threads (the “transsexuals vs slash” business). But they probably think that they can drown out those issues via the volume of data.

        If I were them, I would just have downloaded and used the text of slash fic directly—but I can also see reasons why they wouldn’t want to do that only.

        Obviously, both my ideas and neededalj’s are speculations, but everything they said fits pretty closely with this conceptualization. And I don’t think that any modeling venture can really take place with a bad underlying theory, which they have. So as I said before, I agree with critiques of their behaviour, which reflects an ignorance that is the only thing that can produce many aspects of their theory, but some of the “deeper” technical criticisms seem to be off base.

      2. Asad

        Oh, oops, I was being ambiguous about “same” there. They want “same” in the sense of their (expressed) theory of functional equivalence, but they don’t need the same kind text or the same questions. In fact, the latter would defeat their venture. It would be amusing if the other community they intend to question delivers data that is *actually* similar to the fanfic communities. But they probably have spent some thought on choosing a community that won’t.

        The outcome of this effort is intended to be the validation of a particular machine learning algorithm that reflects their theory about the brain, not conclusions about the communities they are studying. Again, whether this is a valid effort without a better conceptualization of their theory is another question.

  7. Asad

    To give you an example from my own neck of the woods, we know that two children will learn to communicate with each other in “English” even if we expose them to completely different “English” data sets. So those stimuli are not the same, not generated under anything like controlled conditions, but the children acquire something that is the same from them. So the stimuli are functionally equivalent but not “the same”.

    From this point of view, Ogas and Gaddam are trying to build a model of sexual cognition with similar properties. The “circuit” will acquire sexualities that are similar in certain respects but different in ways that children learning English and Japanese are different, because they were exposed to English and Japanese. Again, the flaw is that linguistic cognitive modelers (ie, there are definitely people doing for language what Ogas and Gaddam are trying to do for sexuality) are never really nearly as ambitious as they are, because we don’t really even understand what a neural circuit is

    1. Skud Post author

      Asad, you need to back off. You are dominating the conversation and running off-topic. Please remember that this is a forum for discussing women, feminism and geek culture.

Comments are closed.