Temperament and Philosophy: Two New Papers

Adam Feltz and Edward Cokely have recently co-authored an interesting paper about the relationship between personality type and folk intuitions about free will and moral responsibility.  More specifically, they claim that extraversion is correlated with compatibilist-friendly intuitions.  I have co-authored a commentary on their paper with Trevor Kvaran and Eddy Nahmias.  Since this is the first time that we discuss some of the results from our on-line survey from last summer (see here), readers might find this pair of papers interesting.

Download Extraversion and Compatibilism

Download Temperament and Philosophy

Comments

[Edouard Machery]

Thomas,

Quickly looking at your data, I have the impression that you did not modify your significance level to take into account for the omnibus nature of your test. Most p-values reported (fn 3) are between .01 and .05, and they probably would not be significant if you had lowered the significance level. (But perhaps you did take this into consideration.)

Also, your effect sizes (r square) are quite small.

[tnadelhoffer]

Perhaps this represents a good reason not to look "quickly" :)

On a more serious note, even if you were right--and I still need to talk to Trevor (who ran the stats) to iron out the details--wouldn't that simply further undermine Feltz and Cokely's suggestion that personality traits correlate with philosophical intuitions? We were simply claiming that while there may be some correlations, they aren't as strong or stable as Feltz and Cokely make it seem. I don't see how your criticism would undermine this claim. Am I missing something (which is not unlikely!)?

[Edouard Machery]

No, you are not missing anything. I was obviously not commenting on your discussion of F&C. I am simply interested in your own findings or your lack thereof.

[Bryce Huebner]

Edouard,

I may be making the same mistake by reading quickly, but...

Unless I am missing something in the data, even using a conservative Bonferonni correction to control for the number of comparisons, you still end up with a critical alpha of .0125 (for the 4 t-tests--for the 8 correlations this goes down to .006, leaving only one significant effect). That means that the two cases that looked like they were significant still end up significant. And calculating the r-squared for these cases, it looks like you get a medium effect size by Cohen's standard. Of course, given that we only have the t-tests presented here, rather than the Repeated-Measures ANOVA, I'm not sure that running the t-tests even makes sense. But my guess is that you would find a significant interaction effect that would lead to this analysis (Thomas, is that right, and if so, why not report the results of the Repeated Measures ANOVA?)

[Edouard Machery]

But surely more than 4 tests were run. So 4 is not the appropriate correction. They write: "we examined the correlation between extraversion and the eight individual questions and three composite scores."

And the correlations reported in fn 3 are all around ,15, a small effect. (but you are looking at the t-tests, while I am looking at the correlations.)

[Trevor Kvaran]

Edouard and Bryce,

Thanks for the questions on the analyses we ran. I ran the stats for this, so I'll try to answer them as best I can. I'm still relatively new to running statistics, so I hope the answers will satisfy you.

To the question about adjusting for the total number of tests. Thanks for asking about this because it's something I've struggled with. Maybe this will force me to get straight on what the right answer is. As I understand it, Bonferonni adjustments are fairly controversial. It's not clear that the benefit you get by keeping your potential for type 1 error low is worth the hit you take in potential type 2 error. A lot of the people I've talked to about statistics have told me that bonferroni is too conservative, and probably not a good correction to make, so I didn't do it. My problem has always been that I don't know what the better correction is. Obviously leaving the analyses uncorrected wasn't optimal, and alpha-slippage is a real concern, but I'm not convinced it was a worse choice than any of the corrections I would have known how to make, especially considering the generally deflationary conclusions we draw in our paper.

As to the small effect size problem, I don't exactly know what the issue is. We pretty explicitly point out several times throughout our reply that even when we do find significance, the effects are small. Our commentary is in general a pretty deflationary response to F&C, based in part on the fact that we found small effect sizes.

Bryce's question about why we ran t-tests instead of a repeated-measures ANOVA is a more interesting question to me. The basic answer is that our main goal in this reply was to see whether data that we had would or would not replicate the findings from F&C. To that end, we followed their analysis plan. In their extreme group analyses they ran t-tests to see if the high and low groups were significantly different from each other. Because we wanted our analysis to be as close as possible to theirs, we did the same thing. I honestly never thought about asking the question as a repeated-measures ANOVA, because all we wanted to see was whether our results matched theirs. I think you are right though that that might have been a better way to do the analysis (especially because it could have cut down on the total number of tests we ran and helped with Edouard's alpha-slippage worry).

[Edouard Machery]

Trevor,

Let me say first that I don't think this discussion in any way undermines the interest of your paper.

Second, you write: "Bonferonni adjustments are fairly controversial." This seems false. Adjustments are recommended by every textbook I know of. Second, you can compute the power of your test with or without adjustment to determine whether adjusting the significance level is worthwhile.

[Trevor Kvaran]

Edouard,

I probably wasn't clear enough in the last post. As I understand it, bonferroni adjustments are extremely conservative. They control alpha-slippage, but at a huge cost in potential for type 2 error. Although bonferroni is probably the standard adjustment taught in most textbooks, I have gotten the feeling that it is going out of fashion because it is overly conservative(although you are right that it is probably the most widely used). You might be right though, and I've just gotten my (admittedly limited) advise on stats from an unrepresentative group of people.

Either way, you are right to point out that I should have done some kind of adjustment, and that leaving things uncorrected was not optimal, but I still don't agree that bonferroni would have been the right choice. I'm taking an ANOVA class right now and this topic is coming up soon, so hopefully I'll get my own confusion on the topic resolved before too much longer.

[Jack Yuddles]

Can I strongly recommend that x-philes get up with statistical reform in psych? On the topic of p-values http://www.latrobe.edu.au/psy/cumming/docs/Cumming%20pintervals.pdf is excellent. In short: ditch them. CIs do much better.

[Trevor Kvaran]

Jack,

Thanks for that article. It looks like a really helpful discussion and is exactly the kind of thing I was hoping someone would post here.