People's responses to hypothetical moral scenarios can vary substantially depending on the order in which those scenarios are presented (e.g., Lombrozo 2009). Consider the well-known "Switch" and "Push" versions of The Trolley Problem. In the Switch version, an out-of-control boxcar is headed toward five people whom it will kill if nothing is done. You're standing by a railroad switch, and you can divert the boxcar onto a side-track, saving the five people. However, there's one person on the side-track, who would then be killed. Many respondents will say that there's nothing morally wrong with flipping the switch, killing the one to save the five. Some will even say that you're morally obliged to flip the switch. In the Push version, instead of being able to save the five by flipping a switch, you can do so by pushing a heavy man into the path of the boxcar, killing him but saving the five as his weight slows the boxcar. Despite the surface similarity to the Switch case, most people think it's not okay to push the man.
Here's the order effect: If you present the Push case first, people are much less likely to say it's okay to flip the switch when you then later present the Switch case than if you present the Switch case first. In one study, Fiery Cushman and I found that if we presented Push first, respondents tended to rate the two cases equivalently (on a seven-point scale from "extremely morally good" to "extremely morally bad"). But if we presented Switch first, only about half the respondents rated the scenarios equivalently. Somewhat simplified: People who see Push first will say that it's morally bad to push the man, and then when they see Switch they will say it's similarly bad to flip the switch. People who see Switch first will say it's okay to flip the switch, but then when they see the Push case they don't say "Oh, I guess that's okay too". Rather, they dig in their heels and say that pushing the man is bad despite the superficial similarity to the Switch case, and thus they rate the two scenarios inequivalently.
Strikingly, Fiery and I found that professional philosophers show the same size order effects on their judgment about hypothetical scenarios as do non-philosophers. Even when we restricted our analysis to respondents reporting a PhD in philosophy and an area of specialization or competence in ethics, we found no overall reduction of the magnitude of the order effect. (This research is forthcoming in Mind & Language and has been summarized here.) The Doctrine of the Double Effect is the orthodox (but by no means universally accepted) explanation of why it might be okay to flip the switch but not okay to push the man. According to the Doctrine of the Double Effect, it's worse to harm someone as a means of bringing about a good outcome than it is to harm someone as merely a foreseen side-effect of bringing about a good outcome. Applied to the trolley case, the thought is this: If you flip the switch, the means of saving the five is diverting the boxcar to the side-track, and the death of the one person is just a foreseen side effect. However, if you push the man, killing him is the means of saving the five.
Now maybe this is a sound doctrine, soundly applied, or maybe not. But what Fiery and I did was this: At the end of our experiment, we asked our participants whether they endorsed the Doctrine of the Double Effect. Specifically we asked the following:
Sometimes it is necessary to use one person’s death as a means to saving several more people—killing one helps you accomplish the goal of saving several. Other times one person’s death is a side-effect of saving several more people—the goal of saving several unavoidably ends up killing one as a consequence. Is the first morally better, worse, or the same as the second? [Response options: ‘better’ ‘worse’ or ‘same’]
Non-philosophers' responses to this question were unrelated to the order of the presentation of the scenarios. We suspect that many of them didn't see the connection between this abstract principle and the Push and Switch scenarios presented much earlier in the questionnaire. But philosophers' responses were related to the order of presentation of the Push and Switch scenarios. Specifically, the majority of philosophers (62%) who saw the Switch scenario first endorsed the Doctrine of the Double Effect. However, the doctrine was endorsed only by a minority of philosophers (46%) who saw Push first (p = .02). What seems to have happened is this: By manipulating order of presentation, Fiery and I influenced the likelihood that respondents would rate the scenarios equivalently or inequivalently. We thereby also influenced the likelihood of our philosopher respondents' endorsing a doctrine that appears to justify inequivalent judgments about the scenarios, the Doctrine of the Double Effect. Rather than relying on stable principles to reach judgments about the cases, a certain portion of philosophers appear to have reached their scenario judgments on the basis of covert factors like order of presentation and then endorsed principles only post-hoc as a means of rationalizing their covertly influenced judgments about the specific cases.
Manipulating the order of two pairs of scenarios (a Push-Switch case and a Moral Luck case) appeared to amplify the magnitude of this effect, by pushing philosophers either generally toward or generally against endorsing inequivalency-supporting principles. With two scenario pairs ordered to favor inequivalency, we found 70% of our philosopher respondents endorsing the Doctrine of the Double Effect. With the two pairs ordered to favor equivalency, only 28% endorsed the doctrine (p < .001). This is a very large shift in opinion, given how well-known the doctrine is among philosophers and given that by this point in the questionnaire, all philosophers had viewed all versions of each scenario. We then filtered our results, looking only at respondents reporting a PhD and an area of specialization or competence in ethics, thinking that these high-grade specialists (mostly ethics professors at Leiter-ranked institutions) might have more stable opinions about the Doctrine of the Double Effect. They didn't. When the two scenario pairs were arranged to favor inequivalency, 62% of ethics PhDs endorsed the Doctrine of the Double Effect. When the two pairs were arranged to favor equivalency, 29% endorsed the doctrine (p < .05).
The simplest interpretation of our overall results, across three types of scenarios (Double Effect, Moral Luck, and Action-Omission), is that in cases like these skill in philosophy doesn't manifest as skill in consistently applying explicitly endorsed abstract principles to reach stable judgments about hypothetical scenarios; rather, it manifests more as skill in choosing principles to rationalize, post-hoc, scenario judgments that are driven by the same types of factors that drive non-philosophers' judgments.