Experiment Month Results: Intuitions about Scientific Progress

People typically talk about scientific progress. But what does scientific progress amount to? In the philosophy of science literature, one can find three major accounts (See Darrell Rowbottom and Alexander Bird). The Epistemic (knowledge) view holds that people’s concept of scientific progress involves the accumulation of scientific knowledge. The Semantic (truth) view says that scientific progress involves the accumulation of true scientific beliefs, though not necessarily scientific knowledge. The Functional-Internalist (goal) view states that scientific progress primarily involves achieving either theoretical or technological scientific goals. When arguing for or against these three views, philosophers often rely on intuitions about how much progress people think scientists have made in hypothetical cases relative to the amount of knowledge, truth, or problem-solving outcomes that are generated. Typically, these philosophers assume that ordinary intuitions regarding these factors count as some of the best evidence for their particular approaches. But we might wonder: do actual ordinary intuitions best support the epistemic, semantic, or functional-internalist views?

To find out, Wesley Buckwalter and I ran the following 2x2x2 between-subjects experiment.

We wanted to know if people’s actual intuitions about scientific progress differ depending on the particular types of goals scientists may have [technological (working with “electronic circuit boards”) or theoretical (working with “the Theory of Relativity”)]. We also wanted to know whether or not people thought it mattered if the discoveries the scientists made in our vignettes were justified, since this is the essential difference between the epistemic and the semantic views of scientific progress. To capture the two senses of justification discussed in the literature, we constructed vignettes with and without internal justification, as well as with and without external justification (Download Progress_Survey_Materials). Here is an example of one of the eight possible cases that resulted, involving a group of scientists whose goal is to make technological progress and where the relevant discovery is not internally or externally justified:

A group of scientists at the Institute for Advanced Study has been studying the speed of light. Working with electronic circuit boards, the scientists were trying to find particles that travel faster than light. One day, during an anomalous power surge, the computer program the scientists were using managed to physically accelerate the particles in the circuit boards to speeds faster than that of light. As it turns out, the circuit boards work perfectly, and these faster-than-light particles, called “tachyons,” really were being accelerated. The scientists reviewed the logs immediately, but they could not figure out how the program performed the acceleration.

Then, in a contrasting condition, the scientists were given an internal justification, simply by replacing the last sentence of the case above with:

The scientists reviewed the logs immediately, and they figured out how the program performed the acceleration.

In both cases, the scientific belief that is justified (either internally or externally) or unjustified is that there are particles (namely, tachyons) that can travel faster than light.

After reading just one of these vignettes, participants were then asked, the following question:

On a 7-point scale, 1 meaning “no progress” and 7 meaning “enormous progress,” how much scientific progress has been made in the story above?

We found that when the scientists were justified internally, participants were much more likely to attribute scientific progress (M=6.00, SD=1.21) than when they lacked any justification (M=4.58, SD=1.56).

While this is only 2 of the 8 different combinations of cases given, this tendency was indicative of our general finding. The main result of the study was that internal justification had a large impact on people’s judgments about scientific progress despite having or lacking external justification, or having different types of scientific goals.[1] Alternatively, external justification and goal of discovery seemed to make only a marginal impact on participants’ judgments (in fact the only impact goal type had was an interaction whereby the effect of internal justification was slightly smaller in theoretical cases):

 

 

We’re curious to know what people think about these results. Pace the semantic view of scientific progress mentioned above, it seems like these data are suggesting that internal justification actually plays quite a large role in ordinary judgments about scientific progress. Similarly, while the degree of the effect internal justification had on progress was influenced slightly by the nature of the scientific goal under consideration, results also seem to show that people don’t really care too much about the two types of goals scientists had in particular, so long as internal justification is present.

Of course, these data do not tell us what role justification is really playing in the actual practices of the natural sciences leading to progress, but we do think they may suggest that justification is an important part of ordinary judgments concerning the concept of scientific progress philosophers of science are discussing.

---

[1] A multiple-way between-subjects analysis of variance was conducted to evaluate the effect of internal justification, external justification, and scientific goal on participants’ judgments about scientific progress. We found a main effect for internal justification F (1,389) = 67.34, p <.01, but were not able to detect a main effect for external justification F (1, 389) = .15, p =.699. We also detected an interaction effect between internal justification and scientific goal, F (1,389) = 4.86, p < .05.

Comments

[jonathan weinberg]

Not an experimental question, but some theoretical ones:

(i) is it really true that "Typically, these philosophers assume that ordinary intuitions regarding these factors count as some of the best evidence for their particular approaches"? I would have thought that this would be a case where, at a minimum, we would care not so much about the folk on the whole but rather about the scientifically-well-informed.
(ii) I do worry that a proponent of the semantic view would have a fairly easy response here, which is that your subjects may well have understood the scientists in the no-justification condition as also not having the relevant beliefs. E.g., your subjects might plausibly think that, since "The scientists reviewed the logs immediately, but they could not figure out how the program performed the acceleration", that would mean that probably the scientists don't have the belief that there are tachyons present. So, if there's also no belief in the cases with no internal justification, your results aren't pulling apart the semantic and epistemic views.

[Moti Mizrahi]

Jonathan,

Thanks very much for your comments.

As for (i), I’d like to emphasize that the object of our study is the ordinary concept of scientific progress, not the professional criteria that scientists use to make judgments about progress. Even non-scientists often say things like, “The 17th century was a period of great scientific advancement,” or “We’ve made a lot of progress since the discovery of the circulation of the blood,” etc. We wanted to find out what people mean when they make these judgments.

Having said that, I’m not quite sure what you mean by “scientifically informed.” But if you mean something like “people with advanced degrees in science,” then our sample includes participants with advanced degrees in the natural sciences, social sciences, and engineering. (We asked about academic background in the demographic questionnaire.) Now, I think your comments raise an interesting question: what if we compare the responses of those with a background in science to the responses of those without a background in science, would we find any noticeable differences?

As for (ii), by “the scientist not having the relevant belief,” I suppose you mean that the scientists do not consciously believe at the moment that there are tachyons (although, if asked, they would assent to the proposition that there are tachyons), rather than that the scientists would not assent to the proposition that there are tachyons. For, if you mean the latter, then I find it rather implausible to think that the scientists would refuse to assent to the proposition that there are tachyons, especially if we add that the results were replicated. Now, if you mean the former, i.e., that the scientists dispositionally believe that there are tachyons (that is, if asked, they would assent to the proposition that there are tachyons), then I think that friends of the epistemic view could reply that this case counts as a case of scientific belief, not because any individual scientist believes that there are tachyons, but rather because, once scientific results are established, they become part of the annals of science, as it were, whether anyone in particular believes them or not. In other words, scientific knowledge is collective knowledge and scientific belief is dispositional.

[jonathan weinberg]

Hi Moti,

It might be the case that one is trying to study the ordinary notion of scientific progress, but that nonetheless one might think that only folks with a fairly high degree of scientific literacy would be worth consulting on the matter. E.g., if I want to study, say, the ordinary notion of a novel (and not what highfalutin literary theorists think about the novel), I might still think that the responses of people who pretty much just don't read hardly at all, will not be of much value in that project. The worry is that people with very weak scientific or literary backgrounds would have, at best, partially mastered the concept in question, and that one might be picking up patterns in their answers that are being determined more by their ignorance than by the concept. (One might worry in the cases that you are using that subjects who are unaware of what a big deal the speed of light is in modern physics, just aren't going to be making judgments about these cases that we should care about.)

Regarding (ii) again, the problem is that to pull the semantic and epistemic views apart, you need cases where the belief condition is met but not the justification condition, or vice versa, and preferably both. And it just does not seem to me that any of your materials do this. (Nothing about dispositional vs. occurrent beliefs is coming into the matter.) Perhaps you could explain a bit more explicitly what it is in your materials is supposed to speak to the semantic vs. epistemic debate here.

[Moti Mizrahi]

Thanks very much for the clarifications, Jonathan.

As for (i), our sample includes participants with various backgrounds in the sciences (ranging from a few college courses to Ph.D.). By the way, it seems to me that one does not have to have an advanced degree in science in order to be “scientifically literate.” Consider an autodidact who is an avid reader of popular science journals. Or consider someone like Benjamin Banneker, a self-taught astronomer.

As for (ii), our cases pull the semantic and epistemic views apart by describing a scientific belief that has no justification (either internal or external). We take it that this is the main difference between the semantic and the epistemic views. So, in the case quoted in the post, there is no internal justification because the scientists don’t have any access to what makes the belief that there are tachyons true (the justification for this belief is not recognizable on reflection), and there is no external justification because the belief source is not reliable (the acceleration was the result of an anomalous power surge).

I should also mention that proponents of the epistemic view, like Bird, do not think of knowledge as JTB. Rather, Bird prefers Williamson’s conception of knowledge as a fundamental concept in epistemology that does not admit of an analysis.

[Wesley Buckwalter]

Hey Jonathan, thanks for the comments.

About (i) to get a sense of the dialectic regarding the use of intuitions in the current phil sci debate about the concept of sci progress, perhaps you could just have a look at the relevant papers Moti links above. I think you’ll find (see Rowbottom’s “What scientific progress is not”) that it seems pretty much like business as usual regarding philosophers’ assumptions of ordinary case intuitions. As Moti mentioned, this particular sample was surprisingly diverse qua higher scientific training (for what it’s worth no effects to speak of there), but I’m not sure it would matter all that much for the argument. To my knowledge, nobody is really invoking strict enough scientifically-well-informedness criteria that would disqualify these intuitions [[but not to say that one could or should not argue otherwise independently of this very specific phil sci debate…as I’m sure many readers here will find research about (i) the actual practices across and within scientific fields, and (ii) what expert scientists think about best practices to be much more interesting questions!]]

Now I guess it could be that something not about the phil sci arguments, but rather about our particular cases, that is calling for advanced scientific training (for instance realizing what a big deal those techie sounding particles are) but I find that pretty hard to believe given the current results…

About (ii) I think you’re right that we could have made the belief component a bit more explicit here. Then again…when I read that the scientists could not figure out how the program performed the acceleration it sounds to me a lot like they had to have formed a belief about the acceleration in order to investigate it.

[Darrell Rowbottom]

Just spotted this. A brief comment on (i): personally, I think intuitions are theory-laden (as are the results of thought experiments, in science and philosophy). So I wouldn't accept that just anybody's intuitions were relevant (although as I said in personal correspondence, I'm interested in the results). I could also add that I was happy to use thought experiments, in this particular case, because I was engaged in an argument with Bird (who makes free use of such devices).

(I've no idea about whether the 'typically' statement holds. I am not aware of any discussions of whose intuitions we should be interested in, in the relevant literature. I think you should present evidence in favour of the claim. I would be interested to see it.)

One further thing. Bird's view requires that _no scientific progress whatsoever_ is possible in the absence of (an increase in) knowledge. And my argument is targeted at this claim in particular. Could you clarify what the intuitions are on this? (How many respondents gave a '0' result for the cases where no justification was present?) I'm curious. (Although I wouldn't say this could decide against Bird's view, for reasons explained above, I think you may want to.)

(Of course, the additional tricky bit, with respect to the last paragraph, is whether there is always _some_ increase in knowledge that's implied by the cases you discuss, e.g. about experimental results. As I discuss in my paper, we also have to be careful to distinguish between increases in knowledge and increases in _scientific_ knowledge. It's a messy business.)

[Moti Mizrahi]

Hi Darrell,

Thanks very much for these helpful comments.

I would also like to take this opportunity to thank you for giving us very useful feedback as we were writing our survey materials. Thanks!

Regarding (i), I think Wesley was saying precisely that, to the best of our knowledge, no one in the literature is saying that we should discriminate against certain people’s intuitions, and so we took the liberty of counting all intuitions as equal.

Could you say a bit more about why you think that not anybody’s intuitions are relevant here?

As for your question, by a quick count, 12 participants gave a 1, which means “no progress,” answer.