Validation of Computer Simulations from a Kuhnian Perspective

Contents

4.1 Do computer simulations require a new paradigm of validation?

While Kuhn's theory of scientific revolutions is mainly concerned with the supersession of scientific theories, his concept of paradigms can also be applied to other aspects of scientific practice. For example, it might be applied to changes in the logic of scientific research. The question whether computer simulations bring about (or require) a new kind of research logic is particularly salient, because it has been argued recently that computer simulations somehow blur the line between models and experiments (Winsberg 2009). But if this means that computer simulations are - just like experiments - somehow empirical, the question naturally arises whether the validation of computer simulations can still be understood along the lines of what has earlier been described as classical research logic. Or, if a new paradigm of validation is necessary to assess whether a simulation adequately captures its target system or not?

Before the recent discussion about the relation of simulations and experiments, this question seemed to be rather trivial and its answer obvious: Computers are calculating machines and computer simulations are nothing but programmed mathematical models that run on the computer. Therefore, computer simulations can just like models produce no other than purely inferential knowledge, that is, knowledge that follows deductively from the premises built into the simulation. In particular, computer simulations cannot produce genuine empirical knowledge like experiments or observations can. It is true that computer simulations can produce new knowledge, because they yield logical consequences of the built-in premises that were not formerly known to us (Imbert 2017, sec. 1.3.4). It is also true that computer simulations can - like any model - produce knowledge about empirical reality, because the premises built into them have empirical content and so have their logical consequences. But this is far cry from the empirical knowledge that experiments or observations yield and which - because it is of empirical origin - is genuine. But then computer simulations have just the same epistemic status as theories and models and therefore follow the same research logic and require just the same kind of validation. Now, in order to validate a model or a theory it must be tested empirically, and so must computer simulations.

What I have just described is more or less the picture of computer simulations that was pertaining in the general literature on simulations up to the beginning of the millennium. It had by that time been fleshed out with two distinctions that make the difference between computer simulations and empirical research procedures extraordinarily clear: Firstly, by the distinction of the modus operandi. Is it a formal procedure (computer simulation) or a material process (experiment)? Secondly, by the distinction of their relation to the target system. Accordingly, this relation could be characterized as one of formal similarity (Guala 2002) with the object of the simulation being a representation (Morgan 2003) of the target system or, in the case of experiments, one of material similarity with the object of experimentation being a representative of the target system.

In recent years, however, there has been a persistent discussion among philosophers of science during the course of which the distinction between simulations and experiments has been seriously called into question. Most notably, some authors have claimed that it is impossible to make a sharp distinction between simulations and experiments - at least as far their epistemic reach or inferential power is concerned. (Winsberg 2009, Parker 2009, Morrison 2009, Winsberg 2015). Others have advocated the weaker claim that while there is a distinction between the two categories, the transition between them is smooth and that there are borderline cases for which it is difficult to determine into which category they fall (Morgan 2003).

Now, if this were true, then the generally accepted research logic of empirical science, which relies on the ability to distinguish clearly between empirical observation and theoretical reasoning would find itself in a serious crisis and we would have to expect and, in fact, need to hope for new paradigms of research logic and, in particular, for the validation of computer simulations to emerge.

Figure 1.

However, the case for the non-discriminability of simulations and experiments rests almost entirely on conceptual confusions and an ambiguous use of the term “experiment”. The examples with which supporters of the non-disicriminabilty thesis demonstrate their claim concern almost exclusively atypical kinds of experiments, where the object of experimentation is not really a representative of the target system. For example, Winsberg (2009, 590), discusses “tanks of fluid to learn about astrophysical gas-jets” as an instance of an experiment. But this is an atypical experiment, because the tanks of fluid are not representatives of the target system (astrophysical gas-jets). This kind of experiment is indeed in no better position to produce genuine empirical knowledge about the target system than any computer model. But the fact that there are such atypical experiments does not contradict the fact that there exist real experiments that can produce genuine empirical knowledge about their target system and that this is a feature that distinguishes real experiments from models.

The conceptual confusion that exists in the philosophical discussion about the relation of simulations and experiments can easily be clarified by the schema on figure 1, which depicts the overlap in the use of the words “simulation” and “experiment”. The kind of experiments that Winsberg and other authors advocating the non-discriminability between simulations and experiments discuss over and over again, has been termed “analog simulation” in the schema. As all experiments do, “analog simulations” operate on a material object, but this object does not have a material similarity to its target system and therefore is only a representation, but not a representative of its target system. The latter is required for an experiment to produce genuine empirical knowledge about its target system.

That simulations are not experiments - save for the ambiguity and overlap in the use of words - becomes furthermore clear if we consider the kind of experiments that give rise to anomalies and which in retrospect are declared crucial experiments that decide the choice between conflicting theories. Because the laws of the scientific theories are programmed into computer simulations, they cannot be used to test these very theories. If it really was as difficult to distinguish between simulations and experiments as some philosophers of science believe, then it should - at least in principle - be possible to substitute experiments with simulations in any context.

However, if we draw the demarcation-line between analog simulations and real experiments and not, as the authors advocating the non-discriminability-thesis implicitly do, between computer simulations and analog simulations, then we are able to distinguish clearly those scientific procedures that can generate genuine empirical knowledge about their target system from those that cannot. Simulations and, in particular, computer simulations belong to the latter category and therefore have - with respect to validation - the same epistemic status as theories and models. They need to be validated empirically, but they cannot provide empirical validation.[5]

Summing it up, computer simulations do not break the received paradigm of research logic of empirical science. Therefore, a new paradigm of validation specifically for simulations is not needed.

[5] In simulation-science the term empirical is sometimes used to distinguish simulation and numerical methods from mathematical analysis. (Phelps (2016) is an example of this.) But this is just a different use of words and should not be confused with “empirical” in the sense of being observation-based as the word is understood in the context of empirical science.

t g+ f @