Failure of the Expertise Defense: Persistent Bias in Expert Intuitions

Over the past several years, experimental philosophers have presented several studies indicating that some philosophically relevant intuitions are subject to a host of undesirable biases (order effects, framing effects, and actor-observer differences among others). For example, our research suggests that heritable personality traits predict bias in some fundamental philosophically relevant intuitions (Feltz & Cokely 2008, 2009; Cokely & Feltz, 2009; Feltz, Perez, & Harris, in press; Feltz, Harris, & Perez, 2010). In response to these findings, “philosophical expertise” has been used to shield some parts of standard philosophical practice from the worries presented by experimental philosophers (e.g., Ludwig, 2007; Kauppinen 2007; Horvarth, 2010; Sosa, 2010; Williamson, 2007, 2011). One important part of the “Expertise Defense” is that philosophers are assumed to be relevantly different from the folk (e.g., as a result of their years of training) and consequently philosophers' intuitions shouldn’t display the same (or similar) biases.

 

But more recently, there have been serious concerns raised by experimental philosophers about the Expertise Defense. Some have used indirect strategies suggesting that philosophical expertise is unlike expertise in areas known to result in the relevant differences (e.g., in chess) (Weinberg, Gonnerman, Buckner, & Alexander, 2010 see related discussion here). Others have opted for direct strategies showing that for many important everyday behaviors (e.g., voting, returning library books, showing common courtesy) philosophers often display the same (or similar) biases as the folk (Schwitzgebel 2009; Schwitzgebel & Rust, 2010, 2009; Schwitzgebel & Cushman, in press). In a new paper (Schulz, Cokely, & Feltz, in press), we also adopt the direct strategy and present the first evidence that personality predicts persistent bias in verifiable expert intuitions about free will and moral responsibility. These results suggest that, in at least some important fundamental philosophical debates, the Expertise Defense fails.

 

Comments

[Eddy Nahmias]

Hi Adam, interesting article and results. I especially like the idea of developing skill tests (though a couple of your questions do not seem ideal to test for knowledge of the philosophical debates about FW). Just out of curiosity, can you tell us what the mean responses were to the 3 questions (or at least the composite score means) and/or the percentage of participants who were above and below the midpoint? And is there an easy way to report how big the differences were in mean scores between, e.g., top and bottom groups in warmth or in expertise? (I'm just trying to get a descriptive sense of how large these effects are.)

[Fritz Warfield]

"personality predicts persistent bias in verifiable expert intuitions about free will and moral responsibility."

What reason do you have for thinking that this paper involves a discussion of "verifiable expert intuitions about free will and moral responsible"?

[Edward Cokely]

Hi Eddy,

Thanks for the questions. I guess one efficient answer is to give you our predictive model (i.e., the regression equation), which is as follows:

Strength of Agreement = 3.31 – (.23*Free Will Skill) + (.57*Warmth)

Note: Warmth ranges from 1-5 (5 is highest warmth; average score is 3); Phil Skill ranges from 0-10 (10 is a perfect score, average score was 1.5).

So, if you take someone with the highest Phil Skill score (10) who also has the highest level of warmth (5), on average, their score would roughly = 3.8 (i.e., 3.31 - (.23*10) + (.57*5)), a neutral response. But, if you have someone else with the highest Phil Skill scores (10) and the lowest score on warmth (1), on average, their score would be roughly = 1.5, or somewhere between disagree to strongly disagree.

Basically, our data indicate that both warmth and free will skill move scores around by a bit more than 2 points on each of the three “strength of agreement” questions (strength ranging from 1-7). Personally, I think it’s pretty impressive that both factors are roughly of equal magnitude.

Is this about what you would have predicted?

[Edward Cokely]

Hi Fritz,

Basically, we make this inference for the many reasons outlined in the paper… In short, we assess expert intuitions about free will and moral responsibility based on responses to a validated and reliable psychometric instrument (i.e., a free will skill test). It’s a good first step. But sounds like you have some concerns…

[Joachim Horvath]

For what it's worth, the way in which you characterize the expertise defense in your post is not at all how I would understand it. In particular, its point is not "to shield some parts of standard philosophical practice from the worries presented by experimental philosophers", but rather to defend the legitimacy of that practice against experimentalist calls for its general abandonment or suspension. This is compatible with admitting that experimental evidence may provide defeaters against particular philosophical thought experiments, and even that it might eventually undercut the whole practice. So, in this sense there is no "shielding-off" at all in the expertise defense, and the empirical defeasibility of standard philosophical practice is fully acknowledged. On the other hand, at least I would insist that it requires a lot of very robust empirical evidence to undercut a well-entrenched methodology like standard philosophical practice. There is of course no reason to expect that that practice is free from all of the well-known psychological biases that show up elsewhere in human judgment. But what has to be shown, I think, is that it is in significantly worse shape than other established methodologies in this respect, and this I don't see at all right now. Your own results and the ones you cite may constitute small puzzle pieces for an epistemically possible case against the standard philosophical practice, and the expertise defense in particular, but the way towards making such a case is still a very long one, I would argue...

[Jonathan Weinberg]

"to defend the legitimacy of that practice against experimentalist calls for its general abandonment or suspension." Who do you take to be using x-phi results to call for _that_?

The gloss offered by C&F seems to me accurate; the point of the appeals to expertise offered by their authors is to provide reasons to more or less disregard the x-phi work, by offering a sweeping reason to treat (most of) it as irrelevant.

[Fritz Warfield]

"But sounds like you have some concerns…"

Yes I do. I don't think that those who pass a basic "free will skill test" are the relevant experts for consideration of an appeal to expertise. Passing such a skill test seems to involve displaying a lack of incompetency with the basic concepts, but that's not expertise. Compare: knowledge of how the chess pieces move and the rules of chess is not expertise in chess. (I am not here defending a tight analogy between chess expertise and philosophical expertise; I'm just using the comparison to help express my point about the claim that "experts" are being evaluated in this study.

[Joachim Horvath]

Hi Jonathan,

I don't see your point at all. I cannot speak for other proponents of the expertise defense, of course, but I have explicitly conceded that the results of x-phi can (and sometimes do) provide undermining evidence concerning the standard philosophical practice. Even on a very charitable reading of your comment, this is not "to more or less disregard the x-phi work".

Now, I take it that the standard practice at issue here is mainly the appeal to intuitions and judgments about thought experiment cases. So, if you are right that no experimental philosopher argues for the abandonment or suspension of that practice (which I hesitate to concede), then everyone should be happy with the following claim: Standard philosophical practice can more or less proceed as before the advent of x-phi, except in those cases where sufficiently robust experimental evidence undermines its trustworthiness. So I wonder: Does everyone agree to that? Have we really reached an approximate consensus on that claim? If so, then the expertise defense as I understand it should be completely uncontroversial. But somehow I have a vague feeling that this just cannot be right...

[Edward Cokely]

Hey Fritz,

Perhaps it’s useful to consider that good tests don’t need to provide extensive assessments of underlying cognitive mechanisms of expertise. Rather, valid and reliable predictors of expertise are often sufficient... even if the predictor is only a proxy for the skill of interest. For example, in chess, data show that one of the best predictors of chess skill (e.g., tournament performance) is the number of chess books one owns. So, you don’t need to conduct some in-depth analysis of one’s propensity to commit tactical blunders under pressures in order to predict expertise. Instead, you just need to count their books. Our study leverages a similar approach…

Basically, in efforts to develop a reliable and valid measure of free will skill we conducted the validation study described in section 4.1. Results indicate that our test is a strong predictor of differences in things like relevant books read, courses taken, years studied, and a moderate predictor of intuitions concerning freedom and moral responsibility. According to our data, biases related to heritable personality traits are likely to persist even among those who have spent the most time studying, reading and thinking about the relevant issues.

[jonathan weinberg]

@Joachim - It's the plain meaning in texts like Williamson's, Hales's, and Ludwig's that the point of the move is to allow the armchair practitioner to disregard the experimental work. If you have a more nuanced version you want to defend, that's terrific, but then you need to (and should want to, too, since it makes your arguments better than theirs!) acknowledge your substantial differences with those other authors.

"Now, I take it that the standard practice at issue here is mainly the appeal to intuitions and judgments about thought experiment cases. So, if you are right that no experimental philosopher argues for the abandonment or suspension of that practice (which I hesitate to concede), then everyone should be happy with the following claim: Standard philosophical practice can more or less proceed as before the advent of x-phi, except in those cases where sufficiently robust experimental evidence undermines its trustworthiness." But this is _very_ different from your gloss in the earlier comment! Here's at least two positions that are consistent with rejecting both your earlier gloss, and the quoted claim. First, concerns about the trustworthiness of current armchair practice have been raised that impose an intellectual obligation on practitioners to address those concerns in a substantive way. Second, that those concerns also make very salient the possibility that current practices will likely need to be _modified_ -- perhaps drastically, perhaps not, in some ways by means of expansion (i.e., by adding in x-phi), in other ways by means of curtailment (i.e., by restricting the current wide-open scope of the practive). I do think that the x-phi work should raise the _possibility_ of a curtailment so radical that the practices should basically count as abandoned more than modified, but while I think it is a live possibility, I do not think it is a likely one.)

I can't think of any justification for only a case-by-case restriction, unless one held the very substantial (and implausible) empirical view that each and every individual intuition is produced in a way psychologically distinct from each and every other individual intuition.

@Fritz - Adding on the Edward's excellent point about the operationalization of expertise, I would add that it is also incumbent upon those who favor the expertise response to tell us how _they_ think expertise, in the relevant sense, can be recognized. If it turns out to be something that is either incredibly hard to detect, or possessed by only a small fraction of the professional population, then it is not going to be a notion of philosophical expertise that will be of use in responding to the challenge.

[Joachim Horvath]

Hi Jonathan,

the two positions that you claim are consistent with rejecting what I say above are also consistent with accepting what I say there, it seems to me. Concerning the first, it is possible to address the experimentalist concerns on a meta-level, i.e., by talking about the standard practice, while continuing with that practice as before (of course, if you think that the meta-stuff is actually part of the practice, then this is just a verbal dispute). Concerning the second position, my second gloss was maybe less careful than my first. What I want to say is simply this: Present x-phi evidence with respect to philosophical experts is still so thin that it warrants at most a case-by-case restriction - but of course if more evidence gets accumulated we might eventually draw more general conclusions about how to restrict and/or expand standard practice, and we might even abandon that practice altogether (so I acknowledge your point that these possibilities are made salient by x-phi). But one difference between us might be that I think it requires a lot more, and a lot more robust, evidence to justify any such general restriction and/or expansion. And this relates to your final point: there are often excellent reasons for mere case-by-case restrictions. Suppose we repeat your Truetemp-study on philosophical experts and find that they are equally susceptible to the relevant order effect, and we replicate that result often enough to establish that it is a robust effect. What general lessons could we draw from that? Not that intuitions about cases in general are susceptible to an order effect, because the study shows that some are not. Moreover, from that study we do not learn much about the psychological mechanism(s) that generate these intuitions, even though we can of course make some educated guesses - so I don't quite see the relevance of your remark that there probably is a unified underlying psychological mechanism (which actually seems plausible to me). All we should conclude, _on the basis of that study alone_, is that intuitions about Truetemp-style cases are problematic - or, rather, that they are problematic when such cases are considered in close temporal proximity with certain other cases. So, a seemingly responsible reaction would be: let's not appeal to Truetemp-intuitions anymore (unless we know more about why they vary in the relevant way), or, even more carefully, let's not appeal to them when generated in close temporal proximity with considering (certain) other cases.

[Edward Cokely]

Here is a story from Miller-McCune about our new paper that some may find interesting.

http://www.miller-mccune.com/culture-society/extraverts-more-likely-to-believe-in-free-will-32033/

[Adam Feltz]

Hi Joachim,

Isn't checking to make sure that intuitions are not related to personality in undesirable ways the prudent and intellectually responsible thing to do? It seems to me that the case-by-case restriction is not very responsible – especially when there are documented biases in some expert philosophically relevant intuitions. We know that personality is pervasive and likely to be related to many philosophically relevant intuitions. Now, if I understand your advice, in the absence of evidence that the intuitions I am using are subject to undesirable biases, it's OK for me to continue using those intuitions. But I know that my intuitions may be biased in unwanted ways – and this is not only merely possible, it has a non-trivial chance of being actual. I could simply ignore this worry. But then my subsequent view might be biased in ways I don't want it to be. This worry could have been addressed by checking out those intuitions from the very start. So, the “ignore and hope” advice just doesn't strike me as very good advice.

[Daniele Sgaravatti]

I agree with Joachim that it is at least premature to talk about "the failure of the expertise defense". This does not mean that the results in the paper are not interesting. In fact, I believe they are incredibly interesting. But how we should generalize from these results seems still an open question.
There are two issues: one has to do with the individuation of "the experts", and the other has to do with individuating the relevant class of judgements.

About the first issue, it seems to me that this paper actually improves on the existing literature. The initial point of the expertise defense is that the judgements that are relevant to the philosophical community are those of people who are part of the philosophical community (this is connected with the fact that nowadays philosophers do not typically argue from the premise that "in case C, we/the folk/one would say that P is the case", but rather from the premise that "in case C, P is the case"). People who publish and referee papers in an area are responsible as to which judgements about cases are accepted or at least taken seriously enough to be discussed. If so, the experimental philosopher who wishes to argue for a revision of the practice should give reasons to think that people who publish and referee papers in an area are biased in the relevant way. To give an example, Schwitzgebel and Cushman (in press) does not provide *direct* evidence for that claim, since they take possession of a PhD as the criterion for being an expert. Alas, having a PhD is not the same as publishing, having a job, and, in short, being part of the philosophical community.
This paper improves in this respect: if the skill-test is a reliable predictor of expertise in a broad sense, and if (as one hopes) expertise is correlated with being part of the community, the findings come at least very close to providing a direct reply to the expertise defense.

About the other point Joachim was pressing. I think it is reasonable to move by areas. The capacities that drive judgements about goodness or badness of actions, very roughly speaking, can perhaps be evaluted as a whole, but surely they are very different from the capacities that drive judgements about who knows what. Most if not all of the studies conducted on (alleged) experts have to do with moral philosophy and/or philosophy of action. So, even putting aside for the sake of the argument the first worry entirely, the strongest claim that appears to be justified is that the studies show that there is a problem with appealing to judgements about thought experiments in those areas, or that the expertise defense does not work in that area.

[Joachim Horvath]

Hi Adam,

the intended advice was not meant to be "ignore and hope", but rather something like "continue in the absence of specific reasons to doubt". Human beings are of course subject to all kinds of biases in all areas of cognition, including perception and judgment. So, would your alternative advice also be to check out all of our _judgments_ "from the very start". I hope you agree that this advice could easily lead to a crippling form of skepticism. I think, therefore, that even a non-trivial chance of being subject to bias is not yet enough to warrant such an advice - we need a lot more and a lot more specific evidence for that. In effect, then, the idea is to be as conservative as reasonably possible when it comes to generalizing from your and other peoples experimental results concerning the epistemic credentials of intuitions or other basic forms of human evidence, like perception, memory or introspection. Finally, I agree with Daniele that going by areas of philosophical subject matter is one reasonable suggestion of a constraint on generalization. But we need other constraints as well that tell us when we should regard our intuitions about a certain philosophical subject matter, e.g. morality or action, as discredited - and how long we can still continue on the basis of more local restrictions.

[John Dell]

Fascinating work everyone!

So, it at least seems reasonable to say that there is a fair bit of evidence being gathered that the expertise defense fails.

What evidence is there that the expertise defense succeeds?

From what I can tell, the only evidence that the expertise defense succeeds is founded on the intuitions of supposed philosophical experts. This hardly counts for evidence because it is an argument that is both circular and begs the question. In order to try and cement the notion that philosophical "experts" are not subject to bias, philosophical "experts" argue that because of their supposed expertise the bias does not impact their judgments. In essence, philosophical experts intuitions are not subject to bias because the intuitions (which, of course, often come in the form of an argument) of philosophical experts tell us they are not subject to bias. Yikes!

@Adam, I guess I'm not surprised at the reaction you are getting from some, but the funny thing is, give the nature of your research, you could have just as easily gathered evidence for *not* rejecting the expertise defense. It's just that things did not play-out that way. So really, people on both sides should be thanking you for conducting research that gave equal opportunity for gathering evidence in favor of, or against, the expertise defense. Biases are such powerful things, it will be interesting to watch how those acting under the influence of bias will react should the evidence continue to mount against the viability of the expertise defense.

I can't say enough how much I enjoy this work. There have been some really great x-phi papers across the board lately. I say this a little tongue in cheek, but it is just so intuitive that personality plays a role in how we think about philosophical problems, and also, what philosophical problems we think about!