Boo Surveys?

Brian Weatherson has written a post prompted by the New York Times symposium on experimental philosophy. The post makes a lot of valuable points (the Austin bit is particularly interesting, and not something I’ve thought of), but there’s also a line that touched a pet peeve of mine. So I wrote a comment. Since the line is something I’ve heard around these parts, I decided to reproduce the comment here. Let me know what you think!

It seems that “I like the idea of experimental philosophy, but it just relies too much on the survey method” has become a common refrain in criticisms of experimental philosophy. I’ve always found this line of attack a bit puzzling, or at the very least, imprecise.

As far as I understand, the survey method — in the salient sense — simply involves collecting data by asking for people’s self-reports, rather than, say, by observing and coding their behavior or scanning their brains. (Another sense, in which surveying involves no experimental manipulation, clearly does not apply to many experimental philosophy studies.) This method is commonly employed in social and (some) cognitive psychology. For example, studies in the confabulation literature use the survey method. They ask people to report on their judgments and how they come to those judgments. Of course, the experimental manipulations of those studies are what allows the interpretations of the data that led to the conclusion that people are unreliable in their self-reports about some mental processes.

I take it the thrust of the anti-survey refrain is in fact not about the survey method at all, but about the manipulations and interpretations of some experimental philosophy studies. Fair enough. But not all experimental philosophy studies use the same experimental manipulations and the same approach to interpreting the data. So, it seems to me, specific criticisms of manipulations or approaches to interpretation would be much more helpful than broad ones. At the very least, broad criticisms should identify what the problematic manipulations or approaches to interpretation have in common, and which studies fall prey to these problems. It looks like the experimental philosophy community has done a decent job keeping itself in check on those counts (e.g., this and that) even though progress certainly comes incrementally and slowly.

Moreover, once clarified, the thrust of the anti-survey refrain impacts philosophy’s interactions with empirical disciplines generally, and not just experimental philosophy. Perhaps the stake-manipulation study-designs do not adequately address epistemologists’ concerns and the straightforward inference from folk responses to philosophical conclusion is overly hasty. Even so, these are simply the kind of problems that frequently arise with interactions between philosophy and psychology (and likely other empirical disciplines too). Psychologists designing the experiments could easily, and perhaps even more likely given their lack of conceptual familiarity, miss philosophers’ concerns too. They might not know what the appropriate questions to ask either. Similarly, philosophers could misinterpret psychological findings. Additionally, in my experience, psychologists not infrequently misinterpret their own findings by making stronger conclusions than the data warrants; it would be bad, too, if philosophers were to draw philosophical implications by simply taking psychologists’ words at face value. This is not to say that philosophy ought not interact with empirical disciplines, but just that it’s hard generally. Consequently, I find the singling-out of experimental philosophy in this line of attack puzzling.

There are a lot of good methodological criticisms of specific experimental philosophy studies or sets of studies, including criticisms from experimental philosophers themselves, as noted earlier. As such, let’s lay to rest the anti-survey criticism of experimental philosophy in general. Minimally, it’s bad sloganeering.


  • In the interest of full disclosure, I used to adopt the unfortunate “survey” terminology for some experimental philosophy projects as well. So let this post serve as self-correction.
  • There are some experimental philosophy projects, as well as projects in psychology, that do not involve manipulation. Instead, they might do correlational analyses on people’s self-reports; Chandra Sripada’s recent work using structural equation modeling is one example. Even with these projects, though, what experimental philosophers are doing is not merely polling and taking the majority view, as the survey terminology might suggest.

7 Responses to Boo Surveys?

  1. jdmitrig says:

    Sam, I feel like the characterization of Weatherson’s post you provide isn’t particularly accurate. I think you’re correct (and I think Weatherson can agree) that surveys have been fruitfully used to establish interesting results.

    It seemed to me, however, that Weatherson’s point wasn’t that surveys are never a good way to conduct empirical research, but rather that 1) they aren’t a good way to discover what people actually think about difficult philosophical problems; and 2) even if they were, popular opinion isn’t a good guide to truth, so the results of such surveys hold little philosophical interest.

    And I don’t see how anything you’ve written here bears on either of those points.

    • Jonathan Livengood says:

      I’m not sure that you (or Brian, for that matter) are understanding an important part of Sam’s post, namely that there is an important distinction between pencil and paper methods that make use of manipulations and those that do not (sample-survey methods properly so called).

      Often, experimental philosophy work does not depend on the specific judgments that people make but on the fact that they make different judgments in different conditions. But the two criticisms you mention seem to target sample-survey methods properly so called.

    • Shen-yi Liao says:

      Hey Dmitri,

      I mean to pick up on one line of Weatherson’s post only, but it’s a line I’ve heard frequently. I think I actually agree with Weatherson on the particular substantial criticisms. My point is a partly terminological one, but I think the terminology has implications.

      I am not sure what you mean by surveys “aren’t a good way to discover what people actually think about difficult philosophical problems”. There is a trivial sense that this is true: we should not go out and poll people whether contextualism/four-dimensionalism/imprecise bayesianism is true. That is a point against surveys, in one sense of the term.

      But that’s not the sense of the term people care about because what most experimental philosophers do is not polling. There are then issues about experimental design and interpretation of results. I think specific criticisms on those topics can be persuasive, and they can help experimental philosophers do better. However, I think the anti-survey sentiment speaks nothing to that, because the issues there aren’t about the method that studies use to collect data.

      What I’ve written — you’re right — do not address the design and interpretation issues. I think that’s the job for people whose studies are getting specifically criticized. My point is simply that the anti-survey refrain is misdirected because the use of self-reports is both well-accepted and frequently-employed in psychology and the least of experimental philosophy’s problems (some philosophical, some methodological).

  2. jdmitrig says:

    Jonathan, I take it that you’re saying that Weatherson is wrong to claim that surveys aren’t a good way to discover what people think about difficult philosophical issues because surveys with manipulations *are*, as a matter of fact, a good way to discover what people think about difficult philosophical issues?

    If that’s right, then what about surveys with manipulations makes them a better guide to discovering what people actually think than the ‘sample-survey methods properly so called’?

    • Jonathan Livengood says:

      Bluntly, I’m complaining that “surveys with manipulations” aren’t surveys, and surveys aren’t experiments. I think that terminological confusion here has led to conceptual confusion about the nature of x-phi.

      I think that studies that manipulate features of verbal probes are better at discovering what (or better, how) people think than are sample surveys (or opinion polls), and I think they are significantly better than nothing. What makes them better is that manipulations allow us to see better what about a given probe matters to people. In fact, manipulations are generally more informative than simple observations.

      However, I meant to be disputing the claim that x-phi typically targets ordinary judgments about hard philosophical questions or assumes that popular opinion is a good guide to the truth. I actually agree that there are serious challenges to figuring out how people think by using pencil and paper methods. I tend to think that the objections ought to be raised case-by-case, rather than as a generic criticism of x-phi, and I’m not sure that Weatherson does a great job of providing specific objections. Moreover, I think the kind of challenge Weatherson points to can be met, again on a case-by-case basis.

      Anyway, my claim is that Weatherson paints a common and misleading picture of x-phi, which is also implicated by the two criticisms you give. Namely, people seem to imagine a straw man x-phi that conducts opinion polls and then adopts the majority opinion. But x-phi doesn’t work like that. X-phiers typically manipulate things rather than simply observing them, and they rarely (ever?) simply endorse ordinary judgments. More typically, an x-phier will make a conditional statement like, “If one thinks that folk intuitions are important, then …” and explicitly cite “traditional” philosophers that endorse the antecedent. Or an x-phier will point out that irrelevant things matter to ordinary people, so that what ordinary people say should be *discounted*. Or an x-phier will point out that people in two different populations endorse different claims in some area of philosophy, casting prima facie doubt on appeals to intuitions in that area.

  3. […] comments section to Weatherson’s post goes on for quite a while and is worth a look too.  Go Grue, the University of Michigan philosophy graduate students’ unofficial group blog, took issue […]

  4. […] Boo Surveys? August 20106 comments […]

%d bloggers like this: