The Instability of Philosophers' Judgments about Hypothetical Moral Scenarios

Non-philosophers' judgments about hypothetical philosophical scenarios are often labile and variable. They are subject, for example, to substantial order effects, and they vary by culture and gender. Question: Are professional philosophers' judgments about such hypothetical scenarios particularly better grounded? Jonathan Weinberg and Stephen Stich say no. Timothy Williamson says maybe yes. At stake, perhaps, is the reliability of "armchair philosophy" as a research method.

Fiery Cushman and I have started to examine the question experimentally. We begin by examining order effects on moral judgment: We presented hypothetical moral scenarios, in varying order, to three sets of respondents: professional philosophers, academic non-philosophers, and non-academics. (The professional philosophers had graduate degrees in philosophy and were mostly drawn from Leiter-ranked PhD-granting departments.) Non-philosophers, we suspected, would respond differently to the scenarios depending upon the order of presentation, a sign of instability and unreliability in judgment. Our question was: Would professional philosophers show smaller order effects and thus more stable judgments?

They did not. In fact, the overall trend across our data (marginally statistically significant) was for philosophers to show less stability their responses. This was true even for the subgroup of 91 respondents reporting a PhD in philosophy and a competence or specialization in ethics.

So, for example, the classic trolley problem comes in two versions. In the switch version, a bystander diverts a runaway trolley onto a sidetrack, killing one person on that sidetrack to save five people on the main track. In the push version, a bystander pushes a heavy person into the path of a runaway trolley, preventing the trolley from killing five people further down the track, but killing the heavy person. Now, whether they think such actions are good or bad, the majority of respondents rank the two actions the same when they are presented side-by-side. But order of presentation influences this result: Respondents are substantially more likely to rank the two actions the same when the push version is presented first than when the switch version is presented first. (Why? Intuitions about switch are unstable, and tend to descend to match push, if push is presented first. Intuitions about push are more stable, shifting around less, and thus often will not rise to match switch when switch is presented first.) Non-academics rated switch and push equivalently 53% of the time when switch was presented first and 68% of the time when push was presented first. For academic non-philosophers, the differential was 55% vs. 71%. For professional philosophers, it was 54% vs. 73%, including 50% vs. 75% for the ethics PhD subgroup -- effect sizes well within statistical chance of each other.

Moral luck scenarios (e.g., drunk driver hitting a tree vs. hitting a girl) and action-omission scenarios (e.g., snatching away a drowning person's life vest vs. failing to offer him a life vest) produced similar results. In each case, the order effects were about the same size for philosophers and non-philosophers. In our aggregate measure, philosophers (including ethics PhDs) trended toward showing slightly larger order effects overall than did the two comparison groups.

For further details, see our manuscript here, forthcoming in Mind & Language.

[Cross-posted at The Splintered Mind.]

Comments

[Brandon Terrizzi]

Hello,
I am an undergraduate student at the University of Arizona and I have read this article immediately following completing some required reading on moral realism and the argument from disagreement. One required paper was Andrew Sneddon (2009) about the relevance of empirical evidence to meta-ethical positions. In that paper he advises that the proper targets of empirical inquiry are potential moral experts, but that even if we did have such a data set that they would have to be interpreted within a philosophical framework. If I understand the argument correctly then the philosophical framework employed to do the analysis will most likely already admit to a position on value pluralism.
Although there is no claim above about the status of objective moral truths or value pluralism, do the authors believe that this study addresses any of the arguments put forth by Sneddon or perhaps a similar argument, and if so, how would they likely respond to the claim that their interpretation (assuming they took a meta-ethical position influenced by this data) might be question begging?
I hope that I have not misrepresented Sneddon's case or asked an irrelevant question about this work. Any response is appreciated.

-Brandon T.

[Eric Schwitzgebel]

Thanks for your comment, Brandon!

My understanding is that Sneddon is criticizing arguments for metaethical anti-realism or relativism that employ empirical claims about cross-cultural (and other) disagreement about moral norms. You are right that we don't explicitly endorse this sort of argument, and in fact I am myself a moderate moral realist and moral universalist; but I see how someone fond of the argument Sneddon criticizes might make use of our data.

Surely it is correct that one needs a philosophical framework to evaluate the implications of moral disagreement for metaethical positions. But this is the nature of philosophical inquiry in general. One necessarily has an (implicit or explicit) metaphilosophical frame that guides one's understanding of how to evaluate philosophical claims; and that frame is itself going to be philosophically loaded in a way that likely interacts with the positions one is using the frame to evaluate. I suspect this is part of why philosophy is not as progressive as empirical science.

[Wesley Buckwalter]

Hey Eric and Fiery, interesting paper, and amazing results as always! I was just a little curious about how you might respond to the following worry, which I'm sure you must get frequently from the other side of the aisle when presenting this work.

Granted, these data may fit well with other projects and arguments for dethroning the evidentiary value of intuitions based solely on factors like the bearers' expertise, but I wonder if those bearers are ultimately going to be very convinced that order-effects are themselves reason for alarm in this domain. So for instance, is there reason to think that traditional moral philosophers (as we find them out in the wild!) will frequently be coming across situations such that the relevant (and astonishing) susceptibility to order effects you've discovered here becomes salient, in that it actually *does* play a substantive role in altering the content of the judgments they make amongst all their various professional theoretic pronouncements?

Put another way, the defender of intuition based on expertise might want to bite the bullet and admit that philosophers are in fact totally susceptible to *some* psychological effects they would not reflectively endorse, but their intuitions are still privileged because since philosophers don't regularly confront this kind of manipulation, that particular effect hardly ever(?) influences resulting intuitions in practice. Then, they might be free to refine the notion of expertise at work in whatever epistemic argument for privilege based on some other desiderata.

But maybe the move here just to lean on either i) what the effect tells us about epistemic status of the intuition generating mechanism of both philosophers and non-philosophers in these cases, and/or ii) that other non-philosophically relevant psychological effects have been shown that do influence judgments in practice, and this adds to the chorus against privilege?

[Eric Schwitzgebel]

Thanks, Wesley! Yes, I agree that that's a reaction a defender of philosophical expertise might have, and it's not an unreasonable thought. But I do find the reaction a bit strained, for two reasons:

First, I think philosophers *do* often encounter situations in which order effects might plausibly have large effects on judgment, for example in garden-path style arguments and in articles on the trolley problem or the personal identity problem and other similar issues where there people seem to have a variety of conflicting and unstable intuitions.

And second, somewhat as you suggest at the end: The more types of empirical evidence supporting the idea that philosophers' intuitions about hypothetical scenarios are not especially well grounded, the more general pressure there is against those who think philosophers' intuitions about such scenarios have special epistemic merit.

Purist extremism on either side seems unwarranted: Almost everyone would agree, I think, that philosophers are subject to many of the same epistemic vices as everyone else in some respects and have some special skills in other respects. So the question becomes: What are the *boundaries* of philosophical expertise? Fiery's and my results suggest that those boundaries might be narrower than one would have thought.

[Wesley Buckwalter]

Makes sense, I don't disagree! But I guess just to press a little more on this point, it might be helpful if you can spell out exactly how you see work on garden-path style arguments and in articles on the trolley problem or the personal identity problem coming somewhere close to resembling the conditions of the study under which you get these huge effects. I would have thought, at least at first, that what these philosophers are doing when they are doing professional philosopher stuff in these areas is pretty different. In other words, it seems that order effects go a long way in showing that these intuitions perhaps aren't as well grounded as previously thought in that they can be pushed around in the lab, but why think that that psychological susceptibility is relevant in making whatever philosophers judgments are in these cases less-good given the conditions of their actual research practices?

[Eric Schwitzgebel]

That's a legitimate concern, Wesley. Of course laboratory results and real-world results can be very different. I completely agree with that point, and in fact I would describe myself as, in general, more worried about ecological validity than the average psychologist or empirically-oriented philosopher. These results nonetheless, I think, put pressure on the expert-intuiters view of philosophers and on the view that philosohpers', including ethics PhDs', judgments about scenarios are more grounded in stable general principles that they would endorse than are non-philosophers'. I wouldn't claim that this experiment straightforwardly refutes such views; and in fact I continue to think that those views have some merit. I was surprised by our results. Human beings are complex and one study never shows anything; we need convergent evidence to limn the borders of philosophical expertise.

There might also be more ecologically-valid ways of exploring this issue. I've been thinking about that a bit and would be happy to chat with you about it more if you're interested.

[jonathan weinberg]

As I've told y'all before, Eric, this is such a great study, and I'm highly covetous of y'all's design!

Regarding the last points of the discussion so far, I agree that it's always worth thinking about ecological validity, but I think that it's not such a big question when what one is targeting is philosophical practice -- the differences between what it's like to answer these questions as part of a study, and what it's like to answer them while, say, reading a philosophy paper, are pretty darn close. (Certainly by comparison with, say, participating in these sorts of studies, and making real-life real-time ethical decisions.) If someone has some ideas about specific aspects of philosophical practice that would make a relevant difference, then that could form the basis for an investigatable hypothesis. Do you have any more specific ideas here, Wes, that one could make a survey out of?

I also think that it's best not to think of the general argument as resting on the narrowest interpretation of these sorts of results, but rather taking these sorts of results on the whole as indicative that (i) intuitions are pretty noisy, at least somewhat substantially sensitive to a wide range of contextual and demographic variation, and (ii) we do not at this time have any good reason to think that anything in our training or demeanor as philosophers renders us particularly immune to the effects of those variations.

[Wesley Buckwalter]

Hey Jonathan, so I think I agree with what’s been said about EV, and of course, I definitely agree with several of the relevant conclusions we all have been drawing about expertise and intuition in a bunch of other papers. I thought I would just try to hop on the other side of the fence and try to get inside the opposing mindset regarding this particular paper.

On that note, I was imagining that such a person might want to respond to the present study (though perhaps naively) with something like the following thought. Some philosophers spend their entire careers considering these specific thought experiments. So if they don't want to reflectively endorse this psychological factor, then it’s kind of embarrassing that there was an order effect found here. However, when they sit down and write for instance, decades of papers using them, they engage in a process whereby they consider these cases in all kinds of different ways, times, orders, perspectives, variations, etc, where the intuition from this process ends up getting operationalized. Then, it might be tempting to think that while the present studies got philosophers do the embarrassing thing, the effect is pretty local in that it doesn’t speak to some deeper thing they are supposed to be doing back in the chair over the decades of considering the cases (whatever deeper thing being the "real stuff" of which intuition expertise is constitutive).

So the move (I often overhear others making) would be to grant the point above (i) in that you can *sometimes* get variation, but to argue that point (ii) is orthogonal by suggesting that there are lots noise-causing effects (e.g. the present one being discussed) such that lack of immunity to them doesn't threaten this person's favorite argument supporting expertise.

Of course, other studies might provide evidence for different kinds of variation such that immunization deficiencies seem more dire to the defense the above imaginary interlocutor might want to rely on here, but I was just writing this comment basically wondering about a good way to respond to the above position.

[Eric Schwitzgebel]

@ Jonathan: Thanks and I agree!

@ Wesley: Yes, that does seem to me the heart of the ecological validity concern. One first-line response to that objection is to note that ethics PhDs should already have done that sort of thing, especially with standard trolley examples (which where we actually found them trending toward the largest effect). But if so, it didn't seem to stick with them; their average order effect sizes were just as large. Also, I would figure that as soon as an ethicist sees a push or switch trolley case she must pretty much know what's coming next! It seems on the model you offer that it would be natural to predict smaller order effects in such a case.

That said, I agree that the sort of response you mount is an open possibility. It is also, perhaps, open to empirical exploration, though not (I think) using the present type of design.

[jonathan weinberg]

Eric hit the main points I would want to hit, in response to the kind of argument that Wesley says he's heard. First, it's not a _particularly_ good candidate explanation for the phenomena, since there isn't even any amelioration. (Moreover, I would add to that that in cases where you can show experts doing better in their real professional contexts, you can always point to actual things that they know _how_ to do differently. E.g., a trained physicist can deploy an explicit theory of Newtonian physics to solve a simple balls-in-motion problem, even if it is one that they would give the wrong answer to when answering quickly. I have yet to see any such hypothesis offered as to what expert philosophers are supposed to know how to do, that should shield them from these sorts of effects. Simply knowing that their answers shouldn't be order-dependent isn't enough, of course, since that doesn't tell you _which_ order is the one whose answers you should trust.)

Second, I think people Wes has heard running that argument are often kind of confused as to just what it is that the "restrictionists" need to show. It's enough for our purposes to make it a _very live empirical possibility_ that philosophical deliberations are, in some nontrivial way, influenced by these sorts of factors. As Eric notes in his comment fo 4:50pm, that philosophical expertise _might_ so shield folks from these sorts of effects is merely an empirical hypothesis, and one currently without any actual evidence in its favor. It _could_ turn out true. It would be an excellent hypothesis for further scientific exploration. But philosophers do not any more have the right to just _presuppose_ that it is true.

[Eric Schwitzgebel]

Jonathan: You put the point very nicely. I agree that the expertise hypothesis doesn't have direct, systematic empirical evidence in its favor at this point, but I do think there is some a priori plausibility to the idea that philosophers would profit in some way from extended thought about these types of examples, especially when exposed to them in different contexts and attached to pro-and-con arguments. And surely there is at least philosophical expertise in the formal evaluation of deductive syllogisms (perhaps parallel to your physicists case above). So despite Fiery's and my results and your and Stich's very sensible overviews, I don't feel that I can fault anyone epistemically for still being attracted to the expertise hypothesis, especially if it is suitably modest and curtailed. But I would like to think that the ball is in their court and that they ought at this point to be presenting more empirically-informed arguments to bear.

[jonathan weinberg]

That all sounds right, Eric! I guess I'm not sure who you take to disagree with any of that. I'm not aware of Stich saying anything inconsistent with it. And here, for example, is the closing paragraph of my co-authored paper from Phil Psych on the expertise defense:

"The main point of our paper is to point out that playing the ‘‘expertise card’’ is not at all the trump in this debate that some philosophers have thought. Even if philosophers do, as a matter of fact, turn out to have the right sort of cognitive differences from novices to shield them from the restrictionist’s findings, this isn’t any help to the cathedrists unless they are willing to do a lot more work, and the right kind of work, to show that this really is so. Cathedrists offering the expertise defense have generally seemed to do so in order to close down a line of argument, namely, the restrictionist’s argument from various results of experimental philosophy. Yet it turns out that they have at best only opened up this matter for discussion, and for further scientific investigation. We welcome further attempts to clarify what the nature of any philosophical intuitive expertise might be, and whether and how it might be inculcated, either by our practices as they stand or, perhaps more interestingly, by new practices as they might be developed. That philosophers have no reason to expect now, and from the armchair, that we are intuitive experts of the required sort, is not in itself a reason to think that we could not become, with the help of scientific psychology, the kind of experts that some cathedrists have assumed that we must already be."

I would note that we also say pretty much exactly what you say here about formal logic: "Experts are often such not purely in virtue of their own freestanding cognitive excellences, but because of their possession of and mastery with various external decision aids (and their failure to do so can sharply limit the excellence of their performance; see Shanteau, 1992; Sieck & Arkes, 2005). Where philosophers have something like that, we should expect training to be able to produce an appropriate skill, as with the proof techniques of formal logic as an aid to judgments of validity. But we do not currently have anything of this sort for judgments about cases of knowledge, or personhood, and so on."

Overall, the expertise _hypothesis_ -- that philosophical expertise sufficiently immunizes philosophers from all of the sorts of effects being documented by x-phi -- is surely a hypothesis worthy of much further investigation. But at this time it is, at best, a hypothesis, and one without any presumption in its favor, and is indeed a hypothesis with all evidence that has so far been gathered pointing in the opposite direction. So the expertise _defense_ must, at this point in time, be taken to be shooting blanks, and if they want to shoot with some actual ammunition, they'll have to go do the empirical work to get it. Which is all that we've been arguing in the first place!

[Eric Schwitzgebel]

Right, I wasn't thinking that you or Stich disagreed with those remarks, except maybe a bit in tenor or nuance. I might give a smidgen more weight to your critics' a priori plausibilities, when they're modest enough about it, than I tend to hear in your tone. Admittedly, Fiery and I take a pretty hard tone near the end of our paper too!

[Wesley Buckwalter]

Thanks guys! I think these comments were very helpful in figuring out a little bit about both (i) what to say about one kind of worry about the present study philosophers who radically disagree with ya'll (us) might give, and (ii) perhaps why they might give it.

So just to review: under (i), one very specific thing you could say (without vaguely leaning on other work for variation done elsewhere) is that the present data showing order effects don't seem to be well explained by the "30-years study hypothesis" above because the most famous problems (tend) to have the largest order effects.

Then, under (ii) it could be maybe, as you say Jonathan, due to some misunderstanding of the restrictionist' challenge in lieu of the following way you often put it:

"Overall, the expertise _hypothesis_ -- that philosophical expertise sufficiently immunizes philosophers from all of the sorts of effects being documented by x-phi -- is surely a hypothesis worthy of much further investigation. But at this time it is, at best, a hypothesis, and one without any presumption in its favor, and is indeed a hypothesis with all evidence that has so far been gathered pointing in the opposite direction. So the expertise _defense_ must, at this point in time, be taken to be shooting blanks, and if they want to shoot with some actual ammunition, they'll have to go do the empirical work to get it. Which is all that we've been arguing in the first place!"

I really liked Eric's comment on this. Perhaps the misunderstanding just attributed to our poor imaginary interlocutor above is motivated by (a) aggressive rhetoric (often on our part), but also (b) that the mere *possibility* of the lack of the immunization to the, importantly, ev *relevant* kind of effects is just far far outweighed (though I agree incorrectly) by the a priori plausibility many philosophers find the typical expertise position...in this case specifically instantiated as, "the idea that philosophers would profit in some way from extended thought about these types of examples, especially when exposed to them in different contexts and attached to pro-and-con arguments"

[Eric Schwitzgebel]

That seems a sensible way of summing it up, Wesley.

(I should clarify, though, that although the trend was for philosophers to show the largest effect in the traditional trolley variant of the push/switch case (among the four variants we presented), I don't know that that trend reached statistical significance. So maybe "at least as large" is a good way to describe the relative effect size.)

[jonathan weinberg]

Wes, I think that's _mostly_ right. But I would want to object to one part of it, and substantially clarify another.

The part I'd object to is the bit about aggressive rhetoric. I just don't think it's true that restrictionists have deployed outsizedly aggressive rhetoric in comparison to how folks write in the rest of the discipline on the whole, and indeed our tone is _vastly_ more respectful than the dismissive tone adopted by many (though, thankfully, nowhere near all) of our interlocutors. And in the context of the expertise defense in particular, I'm inclined to say: if years of training in philosophy aren't enough to enable someone to read past some bits of rhetoric here and there, and engage with the substantive claims (like the ones quoted above) -- when this is an explicit part of our training in how to read other philosophers' texts -- then what are the odds that the same training will help people with subtle order effects, etc. that _aren't_ any part of our training?

The part I'd want to clarify is where you say "the mere *possibility* of the lack of the immunization". That's a dangerous way to phrase it, because it invites being read along the lines of the way that, e.g., a Cartesian skeptic offers the mere *possibility* that you're a brain in a vat, or whatever. I chose the phrase "very live empirical possibility" on purpose, and it is meant to signify a substantial threshold that must be crossed. For example, it is a threshold that, I think, was not crossed before people started doing these sorts of studies. Moreover, it is also the case that the greater the evidence for the sensitivities in question, the stronger the challenge is, and so it does matter that at this point in time, the preponderence of the evidence is clearly in favor of the restrictionist's way of seeing things. Finally, note that the very weak existential claim that expertise will "profit in _some_ way" the veteran philosopher is not even close to being strong enough to make a difference here. If expertise offers _some_ degree of immunity, but leaves a huge array of other sensitivities in place (and perhaps even exacerbating others), then the expertise defense would still under such circusmtances be a total failure.