What's wrong with social simulations?

Table of Contents

5 An ideology of modeling

The examples discussed previously indicate that simulation models can be a valuable tool to study some of the possible causes of some social phenomena. However, the examples also show that a) modeling approaches in the social sciences can easily fail to deliver resilient results, that b) social simulations are not yet generally embedded in a research culture where the critical assessment of the (empirical) validity of the simulation models is a salient part of the research process and that c) the significance of pure simulation results is likely to be overrated.

Unsurprisingly, simulation models in the social sciences excel when studying those causes that can be represented by a mathematical model as in the case of Schelling’s neighborhood segregation model. Part of the secret of Schelling’s success is surely that he had a good intuition for picking those example cases where mathematical models really work. But many of the causal connections that are of interest in the social science cannot be described mathematically. For example, the question how the proliferation and easy accessibility of adult content in the internet shapes the attitude of youngsters towards love, sex and relationships, is hardly a question that could be answered with mathematical models. Or, if we want to understand what makes people follow orders to slaughter other people even in contradiction to their acquired moral codes (Browning 1992), then any reasonable answer to this question will hardly have the form of a mathematical model.[10]

Unfortunately, the field of social simulations has by now become so much of a specialized field that modelers are hardly aware of the strong limitations of their approach in comparison with conventional, model-free methods in the social sciences. There is a widespread, though not necessarily always outspoken belief that more or less everything can - somehow - be cast into a simulation model. Part of the reason for this belief may be the fact that with computers the power of modeling techniques has indeed greatly increased. This belief has found explicit expression in Joshua Epstein’s keynote address to the Second World Congress of Social Simulation under the title “Why model?” (Epstein 2008).

In the following I am going to discuss Epstein’s arguments and point out the misconceptions underlying this belief. In my opinion these misconceptions are to no small degree responsible for the misguided practices in the field of social simulations. Epstein sets out by arguing that it is never wrong to model, because – as he believes – there exists only the choice between explicit and implicit models, anyway:

The first question that arises frequently - sometimes innocently and sometimes not - is simply, "Why model?"Imagining a rhetorical (non-innocent) inquisitor, my favorite retort is, "You are a modeler."Anyone who ventures a projection, or imagines how a social dynamic - an epidemic, war, or migration - would unfold is running some model. But typically, it is an implicit model in which the assumptions are hidden, their internal consistency is untested, their logical con- sequences are unknown, and their relation to data is unknown. But, when you close your eyes and imagine an epidemic spreading, or any other social dynamic, you are running some model or other. It is just an implicit model that you haven’t written down (see Epstein 2007).

...

The choice, then, is not whether to build models; it’s whether to build explicit ones. In explicit models, assumptions are laid out in detail, so we can study exactly what they entail. On these assumptions, this sort of thing happens. When you alter the assumptions that is what happens. By writing explicit models, you let others replicate your results. (Epstein 2008, 1.2-1.5)

It is not entirely clear whether Epstein restricts his arguments to projections, but even in this case it is most likely false. It is simply not possible to cast anything that can be described in natural language into the form of a mathematical or computer model. But then we also cannot assume that this must be possible, if projections to the future are concerned. It is of course always commendable to make one’s own assumptions explicit. But this does not require modeling.

In addition, there are certain dangers associated with mathematical and computational modeling:

1. the danger of underrating or ignoring those causal connections that do not lend themselves to formal descriptions.
    
2. the danger of arbitrary ad hoc decisions when modeling causes of which we only have a vague empirical understanding. The necessity to specify everything precisely easily leads to the sin of false precision, which consists in assuming detailed knowledge where in fact there is none.
    
3. the danger of conferring a deceptive impression of understanding even if the model is not validated.
    
4. the shaping and selection of scientific questions by the requirements of modeling, rather than by other, arguably more important, criteria of relevance as, for example, the social impact or relevance for public policy.

That Epstein mentions replicability as another advantage of explicit modeling is ironic given that it is still quite uncommon in published simulation studies to give a reference for the reader to access and replicate the model (as described further above). More worrisome, however, is Epstein’s attitude towards validation:

... I am always amused when these same people challenge me with the question, "Can you validate your model?" The appropriate retort, of course, is, "Can you validate yours?"At least I can write mine down so that it can, in principle, be calibrated to data, if that is what you mean by "validate,"a term I assiduously avoid (good Popperian that I am). (Epstein 2008, 1.4)

Calibration (i.e. fitting a model to data) is of course neither the same nor a proper substitute for validation (testing a model against data), as Epstein knows. Validation in the sense of empirical testing of a model, hypothesis or theory is a common standard in almost all sciences, including those sciences mentioned earlier that usually do not rely on formal models like history, ethnology, sociology, political science. It is obviously not the case that validation presupposes explicit modeling, for otherwise history as an empirical science would be impossible.

Epstein furthermore advances 16 reasons for building models other than prediction (Epstein 2008, 1.9-1.17). None of these reasons is exclusively a reason for employing models, though. The functions, for example, of guiding data collection or discovering new questions can be fulfilled by models and also by any other kind of theoretical reasoning. Nor is it an exclusive virtue of the modeling approach “that it enforces a scientific habit of mind” (Epstein 2008, 1.6). Here Epstein is merely articulating the positivistic stock prejudice of the superiority, if only of a didactic kind, of formal methods. Given what Heath et al. (2009) have found out about the lack of proper validation of many agent-based simulations one might even be inclined to believe the opposite about the simulation method’s aptitude to encourage a scientific habit of mind.

It fits into the picture of a somewhat dogmatic belief in the power of modeling approaches that modelers consider the lack of acceptance of their method often as more of a psychological problem on the side of the recipients to be addressed by better propaganda (Barth et al. 2012, 2.11-2.12, 3.22-3.26), rather than a consequence of the still immature methodological basis of many agent-based simulation studies. This attitude runs the risk of self-deception, because one of the major reasons why non-modelers tend to be skeptical of agent-based simulations is that they perceive such simulations as highly speculative. As we have seen, the skeptics have good reason to do so.

[10] A good discussion of the respective merits and limitations of different research paradigms in the social sciences can be found in Moses/Knutsen (2012).

t g+ f @