Experimental Philosophy Metasurvey Results

[I saw a reference to this very interesting work on Billy Dunaway's website a little while ago, and invited him to give us a post about his work with Anna Edmonds and David Manley.  Which is what he's given us below!  So, although the blog will list my name on this post, the rest of this post is Billy's work.  I hope we can get a good discussion going on it! I do recommend clicking through on the link below, too, and you can also find Billy's contact info here, if you want to be in touch with him privately. -Jonathan] 

Some current experimental philosophy is devoted to conducting surveys among non-philosophers to gather information about their dispositions to apply philosophically relevant concepts. And those who report the results of these surveys sometimes make claims about how surprising these results are to philosophers. (Here is a representative quote: "[W]e think that a critical method for figuring out how human beings think is to go out and actually run systematic empirical studies... Again and again, these investigations have challenged familiar assumptions, showing that people do not actually think about these issues in anything like the way philosophers had assumed." (Nichols and Knobe, "An Experimental Philosophy Manifesto" in Experimental Philosophy, ed. Knobe and Nichols, p. 3)) But whether an empirical result is surprising to a group of people is itself an empirical question, and so we designed a survey of our own to test this.

Our hypothesis was that that philosophers would, for the most part, correctly guess what kind of response non-philosophers would give. This was confirmed by our study. We selected several published surveys of folk subjects, each of which had been claimed in the literature to have surprising results. The surveys we chose cover a variety of philosophical topics: causation, intentionality, and moral responsibility. We asked philosophers to suppose that ordinary, non-philosophical folk are presented with the relevant cases, and to say how they thought the folk would respond. (Subjects were firmly instructed to opt out of a given question if they had prior familiarity with experimental research that might bias their answer.) For each question, at least 77% (and up to 95.8%) of philosophers correctly predicted how the non-philosophers would respond.

A brief overview of the questions from the experimental philosophy literature we asked about and the results from our study are printed below. For a more detailed presentation of the questions we asked (which include verbaitim descriptions of the vignettes from the original studies conducted by experimental philosophers) and the results, go here.

We closed the survey after 200 philosophers reached the end. Below, we provide the percentages for those who chose to answer the question rather than to indicate that they could not provide an unbiased answer.

The first question was about a study in the 2008 paper "Causal Judgment and Moral Judgment: Two Experiments" by Joshua Knobe and Ben Fraser in Moral Psychology, Volume 2: The Cognitive Science of Morality: Intuition and Diversity, ed. Walter Sinnott-Armstrong.

There are two studies in the paper; we asked about the study in which subjects are asked how much they agree with the statements 'Professor Smith caused the problem' and 'the administrative assistant caused the problem' (p. 4). 95.8% of 190 respondents predicted the correct result, that agreement with the first statement would be significantly greater than agreement with the second statement.

The second question was about the famous survey about intentional action in Josh Knobe's 2003 Analysis paper "Intentional Action and Side Effects in Ordinary Language". Of 83 respondents, 83.1% correctly said that subjects would answer 'yes' to the question 'did the chairman intentionally harm the environment?' significantly more often than subjects would answer 'yes' to the question 'did the chairman intentionally help the environment?'

The third and fourth questions came from the 2007 paper by Jonathan Livengood and Edouard Machery, "The Folk Probably Don't Think What You Think They Think: Experiments on Causation by Absence" in Midwest Studies in Philosophy vol. XXXI.

We first asked about the "Broken Rope case" (p. 117): Of 202 responses, 78.2% correctly responded that there would be no significant difference in agreement with the statements 'the rope breaking caused Susan to fall', and 'Susan fell because the rope broke'.

We next asked about the "Unsafe Rope Case" (p. 119): of 198 respondents, 86.4% correctly responded that agreement with 'the rope not breaking caused Susan to reach the rafters' would be significantly lower than agreement with 'Susan reached the rafters because the rope did not break'.

The final two questions came from the study in the 2007 paper "Moral Responsibility and Determinism: the Cognitive Science of Folk Intuitions" by Shaun Nichols and Joshua Knobe.

We first asked how subjects would respond, when presented with the abstract question whether people can be morally responsible in a deterministic universe (pp. 669-670); 77.3% of 163 respondents answered that people would answer 'yes' significantly less often than they would answer 'no'.

We then asked how subjects, when presented with a specific description of a brutal crime in a deterministic universe (p. 670), would respond to the question of whether the person who commits the crime is morally responsible. 83.1% of 160 respondents correctly answered that people would answer 'yes' significantly more often than they would answer 'no'.

Comments

[David Manley]

The paper we are working on will, of course, have some discussion about the significance of these results. It is worth stressing that we do not take these results to provide any special reason to doubt (a) that there are plenty of surprising survey-based x-phi results; (b) that there are plenty of valuable papers reporting non-surprising survey-based x-phi results. The idea is to kick up a discussion about what makes papers in the latter category valuable-- in the draft of the paper itself we have some positive suggestions here-- and perhaps to rethink some of the language that is used in motivating the type of x-phi at issue.

[Jonathan Livengood]

Very cool study. Thanks.

In my paper with Edouard (thanks for the citation!), we started with claims made by Lewis and Beebee that such and so was what ordinary people would say. We then showed that those claims were wrong: people didn't say what they were supposed to say.

I'll admit that I'm surprised that philosophers saw these results coming. And I'll admit that it is a hasty generalization from Lewis thinks that ordinary people will say that p to most philosophers think that ordinary people will say that p. But what really makes me scratch my head here is this: how do claims like those we criticize make it into the literature in the first place if so many philosophers know what ordinary people will say and what ordinary people say runs counter to what the author in this or that paper says ordinary people will say? I mean, shouldn't it have been likely that a referee reading one of these papers thinks, "Hmm ... this guy says that ordinary people will say that p, but that doesn't sound plausible at all!" After all, we're not arguing against no one; we're not presenting study results with no connection to the philosophical literature.

[David Manley]

Hi Jonathan-- great question! Maybe if 3 out of 10 referees share one's bad prediction of what ordinary people will say, one can just shop around until one hits on referees that agree. (This will be harder of course the more referees there are per submission...)

[David Manley]

PS. working title: "The Folk Probably Think What You Think They Think (We Checked)".

[Jonathan Livengood]

Ha! I heartily approve the title!

[David Chalmers]

We shouldn't confuse the question of whether someone believes P after being exposed to certain prompts with the question of whether they believed P before being exposed to those prompts. Someone can believe that knowledge is justified true belief, while being disposed to reject that claim after being exposed to a Gettier prompt. Likewise, someone can believe that folk judgments about intentional actions are insensitive to moral valence, while being disposed to reject that claim after being exposed to a Knobe prompt. (After all, philosophers are people too, and one's own reactions can provide evidence about others' reactions.) It seems to me that the relevant surprisingness claims turn on the pre-exposure beliefs, not the post-exposure beliefs.

[David Manley]

Thanks Dave-- the first part of your comment is absolutely right and is one of the points we bring up wrt the positive value of x-phi papers that report unsurprising survey results. Of course, the method of testing one's own intuitions on minimal pairs (for example) is very armchair if not a priori.

I think a good analogy is the testing of one's own 'language organ' about the grammaticality or felicity of sentences. One can go a pretty long way learning about general facts about one's language by that quasi-empirical method, and one can be certainly be surprised by those facts. Meanwhile testing other native speakers of one's own language will typically not yield results that diverge from one's own intuitions. (That's not to say the latter endeavor is never worthwhile--it can be, but its value is not closely tied to the surprisingness of the results.)

As for your final sentence-- we would only take issue with surprisingness claims aimed at motivating the surveys themselves. And more generally it seemed worth investigating whether the intuitions of philosophers are so 'corrupted' by their theorizing that they are out of touch with ordinary intuitions. Happily, that was not the case when it came to these surveys. (One question we did not study: are theorists who specialize in a particular area more likely to be out of touch about folk intuitions in that area?)

[Mark Phelan]

It's good that: "Subjects were firmly instructed to opt out of a given question if they had prior familiarity with experimental research that might bias their answer." But I worry still if the results might not be partly due to implicit learning. There's a lot of research into how mere exposure to a stimulus can influence future behavior and responses, even in the absence of conscious familiarity with the stimulus. This worry was brought to mind particularly forcefully with your results concerning the Knobe effect. Over 40% of your respondents avowed no familiarity with the Knobe effect...but has anyone in the industrialized world really not had at least mere exposure to the Knobe effect results?! (A bit of hyperbole there, of course...but you get the point.) And all the findings you looked at are by major figures in the movement and were discussed in fairly well-publicized papers. I think what you would really want to do to get at what you want to get at here is ask philosophers about purportedly surprising x-phi results that have not yet been publicized at all. Maybe you could team up with some of the usual authors on this blog to look at whatever work they have in progress?

[Thomas Nadelhoffer]

Nice study! I will try to say more later. For now, I just wanted to extend an invitation to each of the authors to be formal contributors to the blog. If any of you are interested just send me an email and I will send you an invitation (with directions).

[David Rose]

Very interesting study! I think Mark is right that the Knobe effect is pretty well known by now and so perhaps implicit learning can account for some of the results. I wonder though whether philosophers would have, in line with Machery et. al's work, been able to predict cross-cultural variation in response to Kripke's Godel case? I also wonder what philosophers would have said if you asked them how they would respond to the cases. For example, some work by Hitchcock and Knobe on judgments of actual causation suggests that philosophers and non-philosophers make very similar causal attributions (they found this in their hospital scenario). Do you think it might be the case that some philosophers initially consider how they would respond and then go on to say that most other people would respond in a similar fashion (work in social psychology has shown these sorts of generalization effects)? If so, might it be that the "unsurprisingness" is just an artifact of philosophers overgeneralizing from their own responses to these cases?

[APinillos]

I think this is very interesting and I wonder if it has implications that go beyond xphi. I wonder if you would get similar results with experimental data from cognitive psychology (Linda Banker case, Wason selection task, Asian disease case)? If so, whatever conclusion one might want to draw from this meta-survey may also apply to that work.

Also, it would be good if the experiments that were chosen for this meta-survey were selected in some systematic way by people not familiar with the hypothesis (not the authors). This could be done by laying out clear criteria and having people not familiar with the hypothesis choose the studies.

[David Manley]

Thanks for all the comments!

Mark: Yes, implicit learning is a concern, though many of the cases we tested are really not that well known outside of the field. (Remember we had respondents from across the spectrum.) Another concern is that some people may have ignored the heavily stressed and bolded bit in the introduction. Testing pre-publicized surprising is certainly a good suggestion for some follow-up research.

David: On the first point: I have no reason to think that philosophers would be able to predict any of the cross-cultural studies. On the second point: presumably we have to distinguish between a philosopher's first-blush inclination and a philosopher's all-things-considered post-theory judgment about the case. Assuming every philosopher was once a member of the folk, one might suppose that the first thing would be a decent guide to what the folk would say. If that is what is going on, our study suggests that philosophers are still capable of accessing those first-blush inclinations despite being affected by theory. Another related take is that philosophers are still capable of a 'folk-simulation', which might generate only simulated intuitions.

Angel-- good points all, and it was great chatting at the Pacific. I think using other people to select the experiments would have been a good idea. It may not have entirely taken care of your concern, since a key criterion was 'is explicitly claimed to be surprising in the literature', and it would have been hard to avoid at least speculation about the hypothesis on the part of whomever was doing the selecting.

[Zoe Jenkin]

Just curious-has anyone done similar studies with the subjects and subject matter reversed? e.g. Can the folk predict the philosophers' intuitions? Not sure if it would have any first-order implications but it would certainly be interesting.