Sara Konrath and I recently published a paper investigating the Knobe effect using an analysis method called structural equation modeling. The paper came out in Mind and Language and is available here. A really nice summary of the methods and findings from our study is available here.
David Rose, Jonathan Livengood, Justin Sytsma, and Edouard Machery wrote a critique that they posted on this blog earlier, and reposted today. It is available here. Their critique is quite strongly worded, and they claim repeatedly in their paper that ‘Sripada and Konrath’s own data undermines their model’.
Rose et al’s claim is very surprising because 1) the methods Sara and I use are completely standard in psychology – the methods we used are the basis of literally thousands of studies published in major journals such as Psychological Science, JPSP, etc…; and 2) we executed these standard analysis methods without errors, as Rose et al themselves have noted. How can it be that standard analysis methods strongly support our model, yet Rose et al conclude that the data in fact undermines our model?
To address this puzzle, I teamed up with a group of Michigan faculty that includes a social psychologist, two statisticians, one person who is a mathematical psychologist and statistician, and a computer specialist. We sought to examine Rose et al’s statistical tests in greater detail.
In short, our conclusion is that the statistical tests used by Rose et al are clearly invalid, and yield demonstrably false conclusions. There were three main statistical tests used by Rose et al in their critique, and that form the backbone of the three main sections of their paper. These tests were put forward by Rose et al, but are not accepted in the structural equation modeling field. These tests superficially look like accepted tests and this probably explains why Rose et al thought the tests were legitimate to use. But the tests are in fact statistically invalid. We demonstrate this by means of detailed theoretical analyses and Monte Carlo simulations.
Our report is available here. The paper is written in a non-technical style, and I hope that people will take the time to read this.
Recent Comments