Anti-Intellectualism about Knowledge, Action and Stakes

 

In this blog (and also at Certain Doubts) there have been several discussions on whether, or to what extent, folk attributions of knowledge are sensitive to practical interests. One issue raised there is whether the folk are tracking knowledge facts predicted by a new theory or cluster of theories in epistemology known as “Pragmatic Encroachment” (McGrath, Fantl, Stanley, Hawthorne, Weatherson, among others). According to this new approach, knowledge is not purely an intellectual notion. It is infused with pragmatic considerations. Recently, Shawn Simpson and I have been running some experiments to investigate this further. In particular, we are interested in a couple of hypotheses which we think support pragmatic encroachment. (These results further support the work I have done before on the topic, which can be seen here: “Knowledge, Experiments and Practical Interests”, but they may be at conflict (perhaps) with some of the interesting work and discussion carried out by May, Sinnott-Armstrong, Hull, and Zimmerman; Buckwalter; Feltz and Zarpentine and Schaffer and Knobe. For a critical review of some this literature, see DeRose or me.)

 

Hypothesis 1. Imagine there are two agents in very similar situations where P is true and who (a) both have the same amount of evidence for P (or stand in the same "intellectual" relation to P), (b) both believe or accept P, (c) both have the same mistaken opinion concerning what is at stake for them concerning P, but the situations differ in what is actually at stake for the agents (one is in a high stakes situation and the other in a low stakes situation). Hypothesis H1 says there are situations that fit the constraints of the pair just described where people are more likely to attribute knowledge that P to the agent in the low stakes situation than to the agent in the high stakes situation. We gathered some evidence which suggests this hypothesis is true. It is worth noting that previous studies that probed for stakes, including my own, did not quite ensure all of (a-c) above are in place. Shawn and I think that (a-c) are important.

Hypothesis 2. The second hypothesis concerns the connection between knowledge and action. Many defenders of Pragmatic Encroachment have argued for principles such as the following: [ACTION] (for P at play) If X knows that P, then it is proper for X to act on P (Fantl and McGrath). H2 is the hypothesis that the folk attribute knowledge and appraise behavior in accordance with ACTION. Our results also support H2.

 

To test (H1) and (H2), we assigned one of the following two vignettes to workers on Amazon Turk living in the United States:

 

LOW STAKES: Peter is a college student who has entered a contest sponsored by a local bank. His task is to count the coins in a jar. The jar contains 134 coins. Peter mistakenly thinks the contest prize is one hundred dollars. In fact, the prize is just a pair of movie passes for this weekend. Peter wouldn’t want them, however, since he is leaving town this weekend. So nothing bad would happen if Peter doesn’t win the contest.  After counting the coins just once, Peter concludes there are 134 coins in the jar. His friend, who also thinks the prize is one hundred dollars says to Peter “you only counted once, even if there are in fact 134 coins in the jar, you don’t know there are 134 coins in the jar. You should count them again”.

 

HIGH STAKES: Peter is a college student who has entered a contest sponsored by a local bank. His task is to count the coins in a jar. The jar contains 134 coins. Peter mistakenly thinks the contest prize is one hundred dollars. In fact, the prize is $10,000 which Peter really needs. He would use the money to help pay for a life-saving operation for his mother who is sick and cannot afford healthcare! So the stakes are high for Peter since if doesn’t win the contest, his mother could die. After counting the coins just once, Peter concludes there are 134 coins in the jar. His friend, who also thinks the prize is one hundred dollars says to Peter “you only counted once, even if there are in fact 134 coins in the jar, you don’t know there are 134 coins in the jar. You should count them again”.

 

Subjects read the following prompt:

Besides giving Peter advice about what he should do, Peter’s friend also said that Peter doesn’t know something. He said that since Peter only counted the coins once, Peter doesn’t know that there are 134 coins in the jar (even if it turns out there are 134 coins in the jar). We are interested in your opinion about this. To what extent do you agree with the following statement: 
“PETER KNOWS THERE ARE 134 COINS IN THE JAR”

 

Subjects were asked to mark their answers on a 7 point Likert scale with ‘0’= ‘strongly disagree’, 3=Neutral and 6(7)=Strongly agree. We found that there was statistically significant difference t(163)= 2.23, p= .027, between the responses to the High Stakes(M=3.058, SD=1.76) and Low Stakes scenario (M=3.68, SD=1.76), d=.35.  This supports our first hypothesis (H1).

 

To test the second hypothesis (H2), we  asked our subjects from the previous probe whether they also thought that Peter should count the pennies again. Here, they only had three options: NO, NEUTRAL and YES. Coding the NO and NEUTRAL in one category and coding the YES in a second category, we can compare the answers to the knowledge prompt above across these two groups. If people tend to act in accordance to the ACTION principle, we should see that subjects in the YES category are less likely (compared to the other group) to agree with the knowledge statement from the prompt above. In fact this is what we found. YES SHOULD (M=3.1, SD=1.69), NO/NEUTRAL(M=3.7, SD=1.9),  t(163)=1.91, p=.029 (one-tailed), d=.3

 

These are two simple examples of the sorts of experiments Shawn and I have been running in the last couple of months. The other experiments tend to also give us results in this direction, as did the experiments I report on the paper I linked above. We can ask several questions about these modest results. (a) Do these results really support (H1) and/or (H2)?  (b) Do they support the idea that folk ascriptions of knowledge pattern in the direction that pragmatic encroachment theories predict for knowledge? and (3) Do they support the epistemic claim that knowledge is sensitive to stakes in the sense of Pragmatic Encroachment? What do you guys think? Shawn and I would very much appreciate any feedback on any of these issues.

 

Comments

[jonathan weinberg]

Hi Angel,

I love reading watching your research project here continue to unfold! One quick question: it seems that it is pretty likely, given the numbers you reported, that a fair number of your subjects might have given a a moderate to high agreement to the "knows" probe, while still answering "YES" to the "should" probe. Wouldn't the presence of a nontrivial number of such subjects be a problematic _dis_confirming turn of affairs for H2? After all, the intellectualists would surely predict something like your finding in terms of the correlation between the "knows" and "should" answers; they will generally think that the subjects' evaluations of Peter's evidential situation will affect both their attributions of knowledge, and their recommendations of action, but just not via a principle like ACTION. But H2 seems to make a much stronger prediction, namely, that _modulo_ a handful of performance errors, etc. in your sample, there should be basically no one who both attributes knolwedge, but thinks Peter should double-check anyway.

[Veronica Beaudry]

Hi Angel,

While I cannot comment directly on the philosophical implications of your results, I thought I would share my knowledge about asking Likert scale questions and the inherent biases of people who answer them.

These question types are often used in market research and the results of such questions are rarely presented as the mean. The mean tends to hover around the middle point (3-neutral) and/or does not provide the most meaningful way to look at/interpret your data. What is the real value of someone being neutral about two different questions/situations if you are trying to distinguish between the two?

Likert scale data are often analyzed by calculating the % of respondents who selected the top two box (rated it a 6 or 7). You can then also compare that number to those who rated it the middle 3 (3,4, or 5), and bottom 2 (1 or 2). This provides a more 'robust' interpretation of the data as you are focusing on those who have a strong opinion. It may also be interesting to compare those who strongly agree/disagree with H1 and see how they respond in H2. How did you choose a 7-point scale versus a 5 point scale or even a 4 point scale with no neutal option?

Next, there are some inherent biases that people have when answering these types of questions. There have been many papers written about this that might be of interest, a few of which I can direct you to.

Lastly, one limitation of Likert scales is that even though you are asking them to rate on a scale from 1 to 7, you cannot assume that respondents perceive the difference between adjacent levels to be equidistant. Someone who is deciding between a 6 or a 7 may view that difference as much smaller than deciding between a 1 or a 2.

[Noah]

Very interesting article. I continue to be impressed by your data-driven approach to philosophy! Regarding the results you report for H1: I assume that t(163)=2.23, p=0.027 means that you performed a t-test using the values from 163 subjects (split evenly between the two groups) which had a t-statistic=2.23 and p-value=0.027. If this is the case, I'm hesitant to accept that anything significant about H1 can be concluded from these data. While the test results are statistically significant, are they practically significant? That is to say, do you really believe that these two groups are substantially different from one another? If we plot simulated data for the two distributions you describe (as I've done here http://bit.ly/jK7afN), we see that the distributions are highly overlapping. Thus, given a randomly chosen value on the Likert scale (0-7), how well can we predict which group (lowstakes, highstakes) the value was sampled from? Not very well. If it doesn't help us differentiate the groups, what can we really conclude? Making a conclusion from such data is common in the biological sciences, but I consider it a statistical fallacy. Statistical significance does not imply a significant result!

[APinillos]

Hi Jonathan. thanks for your comments. They are really helpful, as usual. So concerning your points about H2. You are right, there are a lot of subjects that think both that Peter should count again and that he knows he counted right. Is this trouble for H2? I think you may be right that it is, but to really be clear on this we need to have some baseline to compare this number to, otherwise we don't have very good evidence against H2. Any ideas? Relatedly, I would be interested in seeing how invariantist type approaches could explain the positive results found here. It might still be that Pragmatic encroachment theories still have a really good explanation if not the best for the results of this experiment. You are right, however, that there is a big worry about this one.

Hi Veronica. Thank you for the comments. What you say makes a lot of sense. What we would like to do is analyze the data in a variety of ways. If we converge on a trend, then we have something. For example, I also ran some chi-square tests (using frequencies for answer options grouped in natural ways). I think this should alleviate *some* of your worries. And I did get similar results.

[Aero]

I stopped my philosophy studies at an undergraduate level, so bear with me if my criticism seems suspect. But from a perspective of someone who has 1) studied philosophy and 2) counted a lot of change, why should there be a second count, or a worry that a second count would produce a different number? Probably the subject stacked or sorted his coins in a pattern that was easy to count and could quickly recognize an error, thus eliminating the need for a superfluous double check. Not sure if this criticism helps (I'm sure it doesn't)!

[APinillos]

Hi Aero. Im not sure I can give you a full answer. In a way, the study aims to answer your question. Certainly, many of our subjects do agree with you that its not the case that Peter should count again in these stories. A lot of people think, however, that Peter should check again. They might be thinking not that a second count will yield a different number, but rather that a second count will increase Peter's confidence or increase the amount of evidence Peter has for his belief that there are 134 coins in the jar. And the thought is that Peter should improve his confidence or evidence before he gives his final answer. One of the hypotheses we are testing (H2) predicts that if you think that Peter should count again, you are less likely to say that Peter knows there are 134 coins in the jar. Our results seems to support that.

[Eddy Nahmias]

Angel and Shawn, I really like these vignettes. To be honest, my own intuitions suggested that you'd get even bigger differences between the cases, especially the "should he count again" question. Maybe your participants are better than I am at keeping his subjective state of mind in mind--he should only count again if *he* thinks there is a lot riding on the outcome. (There's a meta-question lingering: how confident should he be about what the prize really is?)

The other thing I really like is having people rate agreement with a statement that essentially mimics a claim that naturally occurs in the vignette (Peter's friend's claim). That's a nice feature, one that I'd like to see other x-phi experiments use. Are there other examples of experiments that use this feature? (Thomas and I are sort of using it now, since we're having people read fake news articles about scientific discoveries and asking them whether they agree with the scientists' claims about the implications of those discoveries for free will, etc.)

[Wesley Buckwalter]

Hey Angel and Shawn, interesting results here, nice to see this effect finally starting to pan out after all these tests. I don't have any fancy statistical worries, but I was going to ask for a clarification on the scale used in this experiment. Was it that it ran from 0-7? If that were true than of course, it would be an 8 point scale (with no neutral answer). And if so, I'd join Eddy in surprise not only that the differences between high and low were so small, but also just that (depending on your views about psychometrics) people generally tended to disagree/unsure across the entire experiment despite the manipulation.

So regarding your questions at the end of the post, it seems like on the one hand it is nice to get an effect for knowledge attribution by stakes after all these papers of null findings (so we can be more confident that people are considering these cases in the relevant way and so on). But then on the other hand, once we step back and consider the nature of what is found, we see that stakes make a significant but really quite negligible impact on mean agreement here. So do you think that although such results maybe support H1, you would be hesitant to rely too heavily on the nature of that support as evidence for a view of knowledge that said ordinary people consider stakes to be really important?

@Eddy, maybe the credit for this should go to Keith DeRose (paper above) who got us all in the habit of using DVs that are more about judging truth conditions of knowledge statements rather than using the old Only Believes/Really knows dichotomy (though of course perhaps for purposes beyond the scope of this latest work).

[APinillos]

Hi Wesley. thanks for your comments. I don't quite agree that stakes make a "quite negligible impact on mean agreement". I reported on effect size (d=.3) which reveals a medium size effect for the behavioral sciences on Cohen's benchmarks (which are widely used). However, I do agree that the effect is probably not as large as the effect of justification or truth on knowledge (Weinberg has been talking about philosophy effect sizes recently). So perhaps relative to those other features of knowledge, the effect is small--and perhaps this is why it took a while for philosophers to hone in on it. And sorry, the scale was 0-6 with seven options (so there was a neutral).

Hi Eddy, I am very surprised with some of the numbers for the "Should" probe as well. We are trying to investigate this further. For what its worth, we have done some other "should" probes as well and they line up more or less with the ACTION principle (this is not to say there aren't other explanations)

Hi Noah, the "d" statistic basically measures the mean differences in standard deviation units. So the differences between the means for (H1) is roughly 1/3 of a sd. This is considered a medium size difference for behavioral science effects. You are right, however, that we would not be able to make really precise predictions about how people would respond to the probe. But I do not think this is necessary to say we have gathered *some* new evidence for the hypothesis. the significance plus the effect size is enough to give us some evidence. Of course, it isn't conclusive or anything like that. But it is a piece of the puzzle. I think.

[Jason Stanley]

Angel,

These are very interesting results. Chandra Sripada and I have been working on a paper for awhile on this topic. We get an even larger difference between high stakes and low stakes than you do (thus ameliorating Wesley's concerns).

Jonathan - the status of (H2) is independent of Interest Relative Invariantism. For example, Williamson agrees with (H2), but rejects pragmatic encroachment. (H2) is a theoretical principle all the early pragmatic encroachments folks used to argue for pragmatic encroachment, but one needs an additional theoretical argument from there to pragmatic encroachment (something like - (H2), without pragmatic encroachment, yields skepticism. But this argument isn't intuition driven). Still, very good to have intuitive evidence about the knowledge-action principles.

It's worth mentioning that Brian Weatherson, formerly an IRI skeptic, has now written a series of papers defending it. In one of them he shows that there are arguments for IRI that do not depend upon general principles such as (H2). So not only are there advocates of (H2) who reject pragmatic encroachment, but there are advocates of pragmatic encroachment who do not endorse (H2) (I'm not saying Weatherson rejects (H2) - it's just that his argument for pragmatic encroachment doesn't depend on a principle this strong).

Still, my own route to IRI is via principles like (H2), so I'm very happy to see this work.

[jonathan weinberg]

@Angel - "to really be clear on this we need to have some baseline to compare this number to, otherwise we don't have very good evidence against H2. Any ideas?"

Yeah, that's rough. I think that more generally it might be that H2 is not going to be at all easy to confirm from these specific sorts of studies. (It could have been disconfirmed, I suppose, if the data had come in very different, and showed no relationship at all like the one you found.) The best you can do here is find that there is a correlation, but the existence of such correlations is consistent with nearby competitor hypotheses other than H2.

I wonder whether it might be better to look at whether very many subjects find somethiing like an "abominable conjunction" in these sorts of cases -- i.e., ask them whether they find it acceptable to say that "Peter does really know, but he should check anyway." I might just be blanking here, but has anyone done any xphi work on those sorts of questions?

"Relatedly, I would be interested in seeing how invariantist type approaches could explain the positive results found here. It might still be that Pragmatic encroachment theories still have a really good explanation if not the best for the results of this experiment." I think that we won't be in position to make those sorts of inferences to the best explanation until we have a _lot_ more data in, of various sorts (that's one of the reasons I like, as I said, watching the different ways your project is unfolding!). Here's one kind of hypothesis that is friendly to old-fashioned standard views of knowledge: there's going to be variation in your subject population as to just how reliable they take the typical coin-counter to be. Those who have great confidence in coin-countings will be more likely to both grant knowledge, and deny any need for further countings. Those with less of such confidence will think that re-checking may be required both to meet the standards of knowledge, as well as to make it sufficiently rational for him to stop with his counting at some point. So, you'd expect to see a correlation of the sort you report, but it would be explained on this hypothesis in terms of the distinct causal impacts of the variation in reliability estimations on knowledge attributions on the one hand, and rational-action evaluations on the other.

@Jason - What you say there comports well with my understanding of the lay of the land here (which makes me glad, since I'm sure you have a much better grasp of the lays of this land than I do!) I'm worried I'm missing your bigger point in bringing that up here, though, since it does seem to me that Angel expresses his point in an appropriately clear & modest way: "Many defenders of Pragmatic Encroachment have argued for principles such as the following...." That bit of epistemological geography seems accurate enough, too. No?

[Jason Stanley]

Jonathan-

Yes, Angel's phrasing is perfectly accurate.

It will take a bit of ingenuity to write up surveys doing a comprehensive evaluation of (H2). I think that Hawthorne claims in *Knowledge and Lotteries* that"Peter does really know, but he should check anyway" is an abominable conjunction, but I can't find where he does so (certainly he makes this point regularly in conversation). Jessica Brown, in her 2008 "Subject-Sensitive Invariantism and the Knowledge Norm for Practical Reasoning" produces relatively uncontroversial examples in which it's ok to say "I know that p, but I should check anyway". Brown's case is something like the following - consider a surgeon about to amputate a leg - she has reassured herself to the highest possible evidential position that it's the left leg, but her hospital requires people to do an official check on the log that it's the correct leg. In such a case, she can say "I know it's the left leg, but I should check on the log anyway". That seems totally fine. Jennifer Lackey has similar cases.

I think what's going on in Brown's case is that the "should" is not an epistemic "should". It is rather a practical "should" (given the rules or norms in this hospital...). So it's not a counterexample to (H2). But it's going to be very, very tricky to devise a survey that ensures that people only understand the "should" as epistemic in such circumstances.

[Jason Stanley]

Jonathan,

I agree that the kind of confound you raise is a serious worry. But I'm not seeing how it raises a worry for Angel's experimental design, as it bears on (H1). There is no obvious connection between skill at coin-counting in the situation in which high-stakes Peter finds himself. So presumably views about variable reliability in coin-counting will be evenly distributed across low and high stakes. In fact, it looks like Angel was careful to design his experiment with this in mind. What am I missing?

[Jonathan Weinberg]

@Jason - That's totally right -- I was only addressing Angel's H2.

[Wesley Buckwalter]

@Angel, that's right, my comment before regarding the .6 difference at midpoint had more to do with the notion of philosophical effect size that JW has recently been talking about. Though maybe your latest experiment shows that the effect really is there, and its philosophical size just depends on the various other details of the case. So that will be exciting to see, especially in cases that get beyond questions JW was raising about the perceived reliability of counting things or typo checking etc.

So about those various other details, I was also noticing two things about these particular stimuli (though I don't know if changing them would ultimately make a difference in the results or not so just food for thought). First, are participants ever told that the belief in question is actually true? Maybe that is playing some role here. When I run it on myself, I don't find myself wanting to say he knew (and especially so in high stakes over low stakes for some reason) if I have no idea whether he was actually right.

Secondly, I was also noticing that the experiment manipulates stakes, but both cases also make the error possibilities in each case super salient. That's fine of course, but at least part of the debate here about stakes has also been about whether they themselves are some kind of independent fourth factor in knowledge. Though maybe what you have is evidence (with just 2 of the 4 cells of possible combinations representing the factors that all the papers above have been arguing about) that the stakes of the case are actually interacting with error. So maybe it would be interesting to run a 2x2 in these cases that specify the truth in all 4, but then include an error factor alongside the stakes to see if its actually stakes alone doing the work here (even if for purely philosophical purposes of argument some were just interested in getting the relevant effect working based on some combination).

@Jason, cool I'm excited to hear more about the experimental philosophy you've been doing with Chandra.

[Shawn Simpson]

Wesley,

Thank you for your post; we really appreciate it.

Now you asked whether participants are ever told that what the protagonist believes to be the case (e.g., the belief that there are 134 coins in the jar) actually is the case. You're right that not including such information might result in a confound. But the vignette above contains the following statement: "The jar contains 134 coins." In each vignette it's the third sentence. Also, in the three other sets of vignettes we ran we stated somewhere within the text that what the protagonist believes to be the case actually is the case. For example, our water purifier vignettes contain the following: "...Brian believes he wrote down all the steps correctly. And in fact, he did." And our flight attendant vignettes contain the following: "...Jason thinks this name is not on the list, and in fact it is not on the list." I hope this alleviates any worries about the effect leaving out this sort of information might have.

---------------------------------------------

Jason,

I'm glad to hear you've obtained similar results. I, too, am excited to hear about the work you and Chandra have done.

[APinillos]

Jonathan, that is a good suggestion about the H2 experiment. It would be great, however, to have some invariantist account that gives a coordinated explanation to both the H1 and H2 experiments. The reason is that the experiments involve the same subjects and essentially the same stimuli. It could be, for instance, that the preferred invariantist H1 explanation makes a bad prediction about H2 or vice-versa. Wesley, the salience of error account is plausible. Although I like contextualism and think that salience of error can play a role in fixing the semantic content of knowledge attributions, I do not think it can explain the stakes effects. In the paper, I mentioned above "Knowledge, Experiments and Practical Interests", I gave some data to support this idea--so I am not here simply ignoring that possibility. The 2x2 idea would be helpful. I agree.

[Wesley Buckwalter]

@Shawn, oops my bad! I totally missed the truthiness somehow at the beginning of your vignettes.

@Angel, fair enough. As a contextualist one might be completely happy to think that both factors play an important role here, and I take it that many do (though the issue of empirical support for pinning down which or both is exactly what started the entire experimental dialectic). The only point I'd want to make is that theoretical arguments aside, and just looking at the evidence for certain ordinary practices here, strictly speaking the current experimental results seem to only support the idea that stakes are producing this difference in people's judgements in cases where there is the presence of this really high error salience. As you argue, it totally could be that stakes are doing this completely independently of the super high error salience, but since that is one of the crucial factors everyone has been discussing, I just wouldn't want to assume one way or the other without more data. If you agree, than you might want to alter H1 with that one qualification.

[Wesley Buckwalter]

Hey Angel and Shawn, you know, I was reflecting today on these results, and wondering what it was about this experiment particularly such that it was able to detect an effect that maybe other experiments in the same vein could not. Do either of you have any theories about this? I was thinking maybe one hypothesis here is that peoples' degree of sensitivity to stakes as third-person knowledge ascribers (or evaluators of first person statements of others) could somehow be a function of people's perceptions of the subject's 'traditional' epistemic position. That is, maybe in all the other cases tested with nulls (bank, rickety bridge, etc) the strength of the epistemic position people perceived the subject to have (lots of good evidence, reliable and so on) was so strong it was just washing out whatever effect stakes were actually having on people's knowledge judgments. Then in the present case, the effect arises because Peter's position is weak enough here (e.g. mistakes, his friend issuing a denial, only checking once) to see it.

Do you think something like that could be going on? It would be that given the same epistemic position, it's not that high and low stakes make that same case either a clear case of knowing or not knowing (like what was previously said about bank)...but rather that stakes effects show up in different cases whereby one's epistemic situation is much worse than when it is better. Maybe it would be cool to test this too.

[APinillos]

Hi Wesley, I think what you said may well be right. It would be worth checking out. So Weatherson, for example, reminds us that IRI is an existential theory. It doesn't predict that there is a change in knowledge facts whenever there is a corresponding change in stakes. It just says that sometimes there will be a change in knowledge facts when there is a change in stakes. So it could be that stakes only play a role when evidence is low. However, I suspect this wont be consistent with some strong principle connecting knowledge and action. Or if we want to preserve the principles, perhaps stake sensitivity is only detectable in the folk when the evidence is low. When the evidence is high, the effect of stakes is suppressed due to performance error or something else along these lines...But for a very different take on this, in the papers I wrote which I linked above (and also in DeRose), there are some armchair explanations for those null results. The view I find plausible is that in many of those previous studies, since the protagonist was (1) aware of the high stakes (in high stakes), (2) asserted that, despite this, he knew, and (3) the evidence available to him was somewhat underspecified, then it makes sense to think that subjects (assuming they accept that knowledge is sensitive to stakes) construed the amount of evidence the protagonist had available to him in high stakes to be substantial (at any rate, higher than for low stakes). So the perceived level of evidence was not kept the same across the high and low stakes vignettes. This created a confound, or so I speculate.