Paper on Knobe's "side-effect effect"

Hello everyone,

Bertram Malle and I have recently completed a manuscript that analyzes Joshua Knobe's "side-effect effect," the interesting set of results that has generated an extensive debate (to which many of you have contributed).  We believe that the results we provide in our paper can can account for the original effect, as well as numerous replications and extensions.

In particular, we suggest that belief alone (merely knowing about a negative outcome)--in the absence of desire--is insufficient to denote intentionality.  We also show that desire (or pro-attitude) is a strong input to intentionality (and we argue that Knobe's original  "chairman harm" and "chairman help" weren't parallel with respect to the desire component, which likely contributed to the intentionality asymmetry).  Finally, we show that when people are given other options of describing the agent's behavior (instead of a simple yes/no intentionality question), hardly anyone considers the side-effect to be intentional.  Rather, most people say the agent "knowingly" or "willingly" brought it about.  Taken together, once all methodological and conceptual confounds in the extant studies are cleared up, there is no side-effect effect.

Thus, given that people view the side-effects as both unintended and unintentional, we argue that our results vindicate both the Simple View of intentionality, as well as Malle & Knobe's (1997) component model.

You can download the file below, or from my website

Download guglielmo_malle_sideeffects.pdf

We would be delighted to hear your comments on the paper, especially within the next 2 weeks, which is when we plan to submit the manuscript for publication.

Best,
Steve Guglielmo

Comments

[tnadelhoffer]

Steve,

First, welcome aboard. Second, thanks for posting what I take to be both an interesting and important paper. As it stands, I am still buried in exams and papers, so I don't presently have time to give it the attention it deserves--but I nevertheless thought I would add my quick two cents to get the commentary ball rolling. So here they are:

A) Your suggestion that folk intuitions about desire (which you cash out in terms of positive and negative pro-attitudes) are influencing the results is interesting. However, I am unsure how you are able to distinguish the desire component from the character component. For instance, in Study 4, participants were more likely to judge that the mayor brought about the side effect intentionally when you portrayed him as a nasty individual than they were when you portrayed him as a generous fellow. By my lights, this is consistent with my view that judgments about character can drive folk intuitions about the intentionality of side effects. Just as we are quick to blame individuals who we know to be morally suspect, we are similarly hesitant to blame good individuals who knowingly do bad things. In this case, why isn't "foresight + character" enough to influence ascriptions of intentionality?

B) On p.34, you draw the following conclusion: "We have therefore disconfirmed Knobe's (2004) claim that 'people's intuitions as to whether or not a behavior was performed intentionally can be influenced by their beliefs about the moral status of the behavior itself' and Nadelhoffer's (2006b) similar assertion that 'ascriptions of intentional action are often influenced by evaluative considerations." I beg to differ--especially if we limit our attention to your first five studies. As far as I can tell, you have at best shown that the Knobe effect affects fewer people than Knobe and others originally assumed. For even once you and Bertram control for possible confounds, a robust minority--nearly 40%!--of the participants still judge the negative side effect to brought about intentionally. As such, you do not show that desire is necessary for folk ascriptions of intentional action nor do you show that evaluative consideration have no effect. What you have shown, is that the Knobe effect explains less of the data that originally assumed. As far as I can tell, you offer no explanation at all for why so many people still deem the side effect to be brought about intentionally if the component theory adequately captures the folk concept of intentional action. I think you should consider using the method adopted by Nichols and Ulatowski to see why participants answered as they did. Without this sort of analysis, it seems to me you have overstated what your otherwise interesting and important results actually show.

C) It is unclear why you only ran studies that examined the harm condition rather than looking at both harm and help conditions. Knobe's earlier studies were interesting not just because so many people in the harm scenario judged that the chairman harmed the environment intentionally but because so few of the participants in the help conditions had the same intuition. His results were driven by a between subject clash of intuitions--a clash that your studies don't rule out. For instance, if you ran both positive and negative versions of your new studies, you may find that whereas 40% judge that the agent P brought about side effect x intentionally in the harm conditions, only 10% judge that the agent intentionally brought about x in the help condition. If so, then the claim that evaluative considerations explain the difference would obviously still be in play.

D) Just as a favor to me, I was hoping you might explain (in layperson's terms) the analysis of blame and intentionality that you give beginning on the bottom of p. 21. Since this is an aspect of the paper I am especially interested in, I would like to better understand the methods being used here.

E) In Study 6, why did you first give participants the yes/no intentionality question and then give them the four "accurate description" questions? Isn't it likely that they perceived this to be a trick of some sort? After all, you had just asked them whether P brought about x intentionally (and 73% of them said Yes!), and the you give them the "accurate descriptions." I can imagine a participant thinking the "intentional" description is too obvious to be the answer you were looking for. But setting that worry aside, I have a bigger gripe with the conclusions you draw from the results. After all, it is well established that foresight is necessary to folk ascriptions of intentional action--i.e., for any agent P who intentionally x-ed, P foresaw that she was x-ing. Indeed, in one of my earlier studies, when you make it clear that the agent did not foresee x, the ascriptions of intentionality drop to floor level (especially if it is made clear the person's ignorance isn't itself culpable)! But if this is true, it seems like there is an alternative explanation for your results. Perhaps people were simply thinking it more correct to focus on the basic component--namely, foresight--upon which their earlier ascription of intentional action was based.

F) In Study 7, you avoided the issue that I just raised about S6 since this time you did not first ask them the yes/no intentionality question. And while the majority of people picked (4), 20% still picked (3) as the most accurate description--something you do not try to explain. If nothing else, it appears that even when people have several options, 20% still think the most accurate way to describe the CEO's actions are in line with Knobe's earlier findings. But setting that aside, I would like to have seen the following option:

(6) "The CEO intentionally adopted a profit-raising program that he knew would harm the environment. As such, he intentionally harmed the environment."

I suspect that had participants had this option, it would have decreased the number of participants who chose (4)--but this is obviously little more than speculation!

OK, that's it for now. I apologize in advance if my comments and suggestions are a bit inchoate. I will try to post something better worked out in the days to follow. For now, I hope I have minimally given you some food for thought/discussion.

Cheers,
Thomas

[Steve Guglielmo]

Thomas,

Thanks for the welcome and for your thoughtful comments. For being swamped with teaching duties, you were still able to give a very thorough response! Your thoughts and suggestions are much appreciated. I'll try to address each of your main points below.

A) I agree that the line between desire and character may be murky and that we haven't necessarily distinguished between the two. Given our results, it seems doubtful to me that "foresight + character" is enough to drive intentionality. In that condition (the "bad" mayor), still only 51% said the outcome was intentional, and remember that the intentionality question was framed as a "let" ("did the mayor intentionally let the homeless go hungry?"). If it was framed as a doing ("did the mayor intentionally starve the homeless?") I would predict the rates would be much lower. But I don't have that data for this particular scenario. Nonetheless, the "foresight + character" led to intentionality judgments for only half the people, hardly enough to say that these two components are sufficient. But the character manipulation did bump up intentionality judgments (only 26% said the "good" mayor let the homeless go hungry), so it is doing something. But we argue that it does not have a biasing effect on judgments, whereby people automatically elevate their blame judgments for bad individuals. Rather, the effect of bad character on blame seems to be driven by judgments of intention and intentionality (more on this in response to part D below). In other words, people are more likely to believe that a bad individual intended a negative outcome, and therefore are more likely to call it intentional. This seems to be a quite normative process--in fact, people may believe that a bad person is precisely one who intends negative outcomes (or fails to prevent them). So when we (or anyone else) tell our subjects that the agent is bad, we may, thereby, give them reason to believe that the agent intended the negative outcome (and, thus, that it was intentional). This is what our results seem to indicate.

B) I also would say that when looking only at the first 5 studies, we haven't explained away the Knobe effect. However, what drives our conclusion on p.34 is what we find in the final studies, where we show that people don't truly see the outcome as intentional. But clearly, it is not as if the harming was purely accidental--the CEO still did many things intentionally. I would argue that the roughly 40% of people who say "yes" to the yes/no intentionality question are indicating this--they are refusing to say that the harming was completely unintentional. But this doesn't mean that their saying "yes" reflects any sort of bias in their concept. It reflects the fact that saying "it was intentional" does carry pragmatic or conversational meaning (as Adams and Steadman argue) and that people were given insufficient options to express exactly the way in which they interpreted the behavior.

C) Comparison between the negative and positive conditions is certainly a useful and important undertaking. We are currently looking more closely at the positive cases to figure out what exactly is going on there. But I don't see it as a problem that we omitted them from this paper. In Knobe's original results, the negative case was really the puzzling one. The positive case conformed to what most accounts would predict--without desire or intention, the side-effect is not intentional. So the negative case is the one that is particularly important to explain: why did people judge the outcome intentional in that condition? You suggest running the positive versions of the vignettes we ran in our paper, which might reveal that, say, 10% of people view the outcome as intentional, still revealing a negative-positive asymmetry. But we argue that the positive version has already been run--this was the one that Knobe ran in the original study. The chairman in the "help" version lacked any desire toward the positive outcome ("I don't care at all about helping the environment."). What we did is make the negative version parallel in that respect, so that the chairman has no desire to harm the environment ("it would be unfortunate if the environment got harmed..."). Although the two conditions were linguistically parallel in the original study ("I don't care at all about X"), we argue that desire is implied when one "doesn't care at all" about bringing about negative side-effects (but not so for positive ones). Once we equate the two conditions by removing the desire in the negative case, intentionality drops severely down to 39%. Cushman and Mele (28%) and Phelan and Sarkissian (28%) found similar low rates when removing desire. We are now in the range of Knobe's "help" condition and have no reason to believe that any negative-positive asymmetry remains.

D) The covariance structure analysis (also called structural equation modeling: SEM) is like a glorified multiple regression analysis. In multiple regression, one can examine a single dependent variable that is predicted by one or more predictor variables. Using the models depicted in Figure 2 as an example, a multiple regression can only handle arrows going into a single box. Only one variable is allowed to be predicted at a time. The SEM approach allows one to predict many variables at once (in SEM terms, one can have multiple "endogenous" variables). Thus, one can test a true model of the way in which many variables might be related to one another. Finally, whereas multiple regression focuses primarily on the significance of individual predictors (i.e., is each individual line needed?), the SEM approach also can test the overall "fit" of the model. As a whole, how well can the model account for the actual data?

The two models that we tested on pp. 20-21 are shown in Figure 2. The first model specifies attitude as a predictor of intention and blame (with the expectation that a negative attitude is associated with more frequent "yes" responses to the intention question and greater blame), intention as a predictor of intentionality ("yes" to intention predicts "yes" to intentionality), and intentionality as a predictor of blame ("yes" to intentionality predicts greater blame). The first three arrows are the same in the second model, but the relationship between blame and intentionality is reversed. In the second model, blame predicts intentionality (greater blame predicts a "yes" response). Although every arrow was significant in both models (again, this is basically what a multiple regression would indicate), other indicators suggested that the first model fit well whereas the second did not. Given that the only difference between the two models was the direction of the relationship between intentionality and blame, this provides some evidence (although, admittedly, certainly not fully conclusive) that intentionality drives blame, rather than vice versa.

E) As you point out in F), I think any concern here is eliminated by the results of Study 7, where we omitted the yes/no intentionality question altogether. The results of Study 7 were strikingly similar to those of Study 6 (and its replication), which suggests that people didn't feel tricked when the yes/no question was included first.

F) I'd be curious to see what people would think about the option you mention ("The CEO intentionally adopted a profit-raising program that he knew would harm the environment. As such, he intentionally harmed the environment."). Our data can't really address this option, but they may indirectly speak against it. In Study 7, people could indicate the two most accurate descriptions. If your new option is appealing for people, one might predict that subjects would endorse option 4 ("intentionally adopted profit-raising program that he knew would harm") as most accurate, followed by option 1 ("intentionally harmed environment") as the second most accurate. In fact, only 2 out of 16 people who chose option 4 as most accurate then chose option 1 as second most accurate. Indeed, this study had a small sample size, but this ratio is not very compelling.

But you raise another interesting concern--that the 20% who chose option 3 ("intentionally adopted an environment-harming and profit-raising program") might be displaying some form of the Knobe effect. I have a couple thoughts about this. First, even if this finding reflect some form of the Knobe effect, it seems to be a decidedly weaker effect than what has been argued by Knobe, you, and others. Only 20% of people choose this option. Moreover, even among this 20%, the choice that they select indicates that they may not truly see the side-effect itself as intentional--they are saying that adopting the harming program was intentional, not that the harming itself was intentional. Second, it's not clear to me that this 20%--even if we grant that they do truly see the side-effect as intentional--indicates any presence of the Knobe effect. As Knobe (2004) and McCann (2005) showed, somewhere around 20-30% of people believe that the chairman actually intended to harm the environment. It's quite possible that the 20% of people who see the side-effect as intentional (if, in fact, they really do see it as intentional) are precisely the ones who also think it was intended. If this is the case, then it wouldn't be a Knobe effect finding--the Knobe effect exists when people say the outcome was intentional despite being unintended. Given that a sizeable percentage of people do infer intention, I'm not too worried about the 20% of people who checked option 3 (although we didn't assess intention judgments so I cannot demonstrate empirically that this subset did infer intention). It's still clear that the overwhelming majority see the behavior as knowingly allowing the negative side-effect to occur.

Thanks for starting the discussion on this and, again, thanks for your thoughtful and thorough comments!

Steve

[jonathan weinberg]

Neat studies! I like the stuff about the desire component in particular, and I'm also really glad to see how this research is stretching in new methodological directions.

Nonetheless, I'm not sure that you really have a right to an inference to anything like this: "what drives our conclusion on p.34 is what we find in the final studies, where we show that people don't truly see the outcome as intentional." You have asked people for their preference among a number of different formulations, but it is entirely consistent with those preferences that they still find a large range of them -- and, in particular, the "intentionally" ones -- semantically correct. For example, given the range of options and the comparative judgment that has been asked for, one might prefer a form that looks less likely to be possibly misconstrued over another form that might possibly give a wrong impression. "X intentionally phi-ed" leaves room for accidentally implying, perhaps via pragmatics, that X had the phi-ing as a goal. So one might disprefer that formulation, even while finding it strictu dictu correct. Fwiw, I think that's how I am operating here: I would make similar top choices as your subjects did, but I find all of (1) - (5) on experiment 7 true.

So, the 'bestness' aspect of your task may be getting in the way here. Knobe et omnii alii don't need that the "intentionally harmed" formulation is the best in the harm scenario -- they just need that it is correct there. So it would be better, I think, would be to give the subjects a similar range of options, and then just ask for independent correctness ratings on all of them. You'll probably want to include materials or instructions that make it clear that more than one of the options may be equally correct; and at least one option that is substantively incorrect, as a comprehension/compliance check.

[Joshua Knobe]

Hi Steve and Bertram,

I just wanted to express my agreement with two of Jonathan's comments above -- first, that these are really neat studies that provide a lot of interesting new insight into the phenomena; second, that it is not quite clear how to interpret people's willingness to choose one description over another.

So, for example, suppose someone gave me the sentences:

(a) John Lennon was a singer and songwriter for the Beatles.

(b) John Lennon was a rhythm guitarist for the Beatles.

If I were asked which of these sentences most accurately captured the nature of Lennon's position, I would definitely choose (a). But that does not mean that I think (b) is in any way mistaken. It could be that I think (b) is perfectly correct but that (a) is still the more revealing description here.

Now suppose we instead give people the sentences:

(a) He harmed the environment intentionally.

(b) He harmed the environment knowingly.

People might say that (b) is the more revealing description, but does that mean they think (a) is in any way incorrect or inaccurate?

[Steve Guglielmo]

Hi Jonathan & Joshua,

Thanks for your comments. I agree with the points that you both make--namely that people may still find the "intentionally" phrasing to be correct even if they don't endorse it as the best description. The scenarios you propose could provide some interesting insights into people's thoughts about this.

The fact that 70%+ of people say Yes to the "intentionally" question when given dichotomous response options makes it seem likely that people would still say this phrasing is correct if asked to provide a correctness rating. For sure, correctness ratings would exceed the midpoint, since more than 50% say yes to the Yes/No question.

But the correctness question may suffer from the same difficulty that the Yes/No question does. By saying that the "intentionally" phrasing is incorrect, people may feel that they are denying intentionality outright--that the harming was unintentional. But clearly many things were intentional (adopting the program, violating the norm of preventing harm), so it is awkward to say that the harming was unintentional.

It is revealing, then, that the "intentionally" option is by far the least-chosen option when people are given a number of different descriptors (like "knowingly" and "willingly"). This suggests that viewing the side effect itself as intentional is not the primary way (or secondary way, etc.) that people interpret the behavior.

So if we force people to specifically answer the intentionality question (either with a Yes/No response or by asking them to provide a correctness rating), people will say it was intentional: this response is more correct than saying it was unintentional. But once we free people from answering this particular question, we see that they naturally interpret the behavior in a very different way than "the side-effect was intentional."

It's evident that there are two different question/processes of interest here: one that's interested in whether people make a particular behavioral inference; one that's interested in which inference people make naturally. Both are important questions, but we may find that they don't always provide identical answers--the interesting path is to continue to figure out why this is the case.

Steve

[jonathan weinberg]

"But clearly many things were intentional (adopting the program, violating the norm of preventing harm), so it is awkward to say that the harming was unintentional." It seems to me that, in a study revised along the lines that Joshua & I are suggesting, this would be easy to address -- just make "intentionally adopted the program" one of the other probes.

I think you're conflating two senses of "if we force people to specifically answer the intentionality question" here. One sense of it is that people have to make a determination of the intentionality question, and can't just ignore it in preference of some other formulation. Another sense of it is that people are only given an intentionality question to register their overall take on the scenario. The latter sense is indeed a kind of potential confound for many of the standard side-effect-effect studies, and it's a great thing for you to be looking into. But avoiding such "forcing" in the latter sense just does not entail avoiding it in the former sense. And if we want to understand people's folk theories (or concepts, or whatever) of intentional action, then it seems to me that just such forcing may be required.

Another way of putting this worry is that in your experiments you're in danger of an illicit forcing as well -- subjects are forced to pick their optimal formulation, or top two, and not allowed to register a broader agreement with other items. This forcing can produce a distortion of its own, not entirely disanalogous with the one you're worried about prior studies in this literature involving.