[cross-posted at: http://experimentalphilosophy.typepad.com/experimental_philosophy/2011/05/experimental-philosophy-metasurvey-results.html. This is a summary of the results of a meta-survey about experimental philosophy conducted by myself (Billy Dunaway), Anna Edmonds and David Manley.]
Some current experimental philosophy is devoted to conducting surveys among non-philosophers to gather information about their dispositions to apply philosophically relevant concepts. And those who report the results of these surveys sometimes make claims about how surprising these results are to philosophers. (Here is a representative quote: “[W]e think that a critical method for figuring out how human beings think is to go out and actually run systematic empirical studies… Again and again, these investigations have challenged familiar assumptions, showing that people do not actually think about these issues in anything like the way philosophers had assumed.” (Nichols and Knobe, “An Experimental Philosophy Manifesto” in Experimental Philosophy, ed. Knobe and Nichols, p. 3)) But whether an empirical result is surprising to a group of people is itself an empirical question, and so we designed a survey of our own to test this.
Our hypothesis was that that philosophers would, for the most part, correctly guess what kind of response non-philosophers would give. This was confirmed by our study. We selected several published surveys of folk subjects, each of which had been claimed in the literature to have surprising results. The surveys we chose cover a variety of philosophical topics: causation, intentionality, and moral responsibility. We asked philosophers to suppose that ordinary, non-philosophical folk are presented with the relevant cases, and to say how they thought the folk would respond. (Subjects were firmly instructed to opt out of a given question if they had prior familiarity with experimental research that might bias their answer.) For each question, at least 77% (and up to 95.8%) of philosophers correctly predicted how the non-philosophers would respond.
A brief overview of the questions from the experimental philosophy literature we asked about and the results from our study are printed below. For a more detailed presentation of the questions we asked (which include verbaitim descriptions of the vignettes from the original studies conducted by experimental philosophers) and the results, go here:
We closed the survey after 200 philosophers reached the end. Below, we provide the percentages for those who chose to answer the question rather than to indicate that they could not provide an unbiased answer.
The first question was about a study in the 2008 paper “Causal Judgment and Moral Judgment: Two Experiments” by Joshua Knobe and Ben Fraser in Moral Psychology, Volume 2: The Cognitive Science of Morality: Intuition and Diversity, ed. Walter Sinnott-Armstrong.
There are two studies in the paper; we asked about the study in which subjects are asked how much they agree with the statements ‘Professor Smith caused the problem’ and ‘the administrative assistant caused the problem’ (p. 4). 95.8% of 190 respondents predicted the correct result, that agreement with the first statement would be significantly greater than agreement with the second statement.
The second question was about the famous survey about intentional action in Josh Knobe’s 2003 Analysis paper “Intentional Action and Side Effects in Ordinary Language”. Of 83 respondents, 83.1% correctly said that subjects would answer ‘yes’ to the question ‘did the chairman intentionally harm the environment?’ significantly more often than subjects would answer ‘yes’ to the question ‘did the chairman intentionally help the environment?’
The third and fourth questions came from the 2007 paper by Jonathan Livengood and Edouard Machery, “The Folk Probably Don’t Think What You Think They Think: Experiments on Causation by Absence” in Midwest Studies in Philosophy vol. XXXI.
We first asked about the “Broken Rope case” (p. 117): Of 202 responses, 78.2% correctly responded that there would be no significant difference in agreement with the statements ‘the rope breaking caused Susan to fall’, and ‘Susan fell because the rope broke’.
We next asked about the “Unsafe Rope Case” (p. 119): of 198 respondents, 86.4% correctly responded that agreement with ‘the rope not breaking caused Susan to reach the rafters’ would be significantly lower than agreement with ‘Susan reached the rafters because the rope did not break’.
The final two questions came from the study in the 2007 paper “Moral Responsibility and Determinism: the Cognitive Science of Folk Intuitions” by Shaun Nichols and Joshua Knobe.
We first asked how subjects would respond, when presented with the abstract question whether people can be morally responsible in a deterministic universe (pp. 669-670); 77.3% of 163 respondents answered that people would answer ‘yes’ significantly less often than they would answer ‘no’.
We then asked how subjects, when presented with a specific description of a brutal crime in a deterministic universe (p. 670), would respond to the question of whether the person who commits the crime is morally responsible. 83.1% of 160 respondents correctly answered that people would answer ‘yes’ significantly more often than they would answer ‘no’.