Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns

Models of emergent phenomena are designed to provide an explanation to global-scale phenomena from local-scale processes. Model validation is commonly done by verifying that the model is able to reproduce the patterns to be explained. We argue that robust validation must not only be based on corroboration, but also on attempting to falsify the model, i.e. making sure that the model behaves soundly for any reasonable input and parameter values. We propose an open-ended evolutionary method based on Novelty Search to look for the diverse patterns a model can produce. The Pattern Space Exploration method was tested on a model of collective motion and compared to three common a priori sampling experiment designs. The method successfully discovered all known qualitatively different kinds of collective motion, and performed much better than the a priori sampling methods. The method was then applied to a case study of city system dynamics to explore the model’s predicted values of city hierarchisation and population growth. This case study showed that the method can provide insights on potential predictive scenarios as well as falsifiers of the model when the simulated dynamics are highly unrealistic.


Entities , State variables , Scale
The system is made of 1145 cities of more than 10 000 inhabitants in 1959, localised via actual latitudes and longitudes in former Soviet Union ( There is only one type of agent : urban agglomerations (we will use the shorter term city henceforward), who are characterised by a unique low-level variable to be confronted with empirical data : their population. From this single variable are derived other economic variables, involved in the interactions mechanisms : -wealth -supply -demand In the final stage of development presented in this paper , one more low level variable is computed for each city : TotalOverflowRatio. Observed State variables includes, but are not limited to, the total urban growth, the rank-size distribution shape of the simulated cities, and the diversity of their trajectories

Processes overview and scheduling
Time is modelled as discrete steps, each of which representing a time period of one year, during which interactions occur in a synchronous way.
At each step : 1. each city updates its economic variables according to its population (supply, demand). 2. each city interacts with the others according to the intensity of their interaction potential. For two distinct cities, A and B, an interaction consists in confronting A's supply to B's demand, resulting in a transaction of goods sent from A to B. 3. each city updates its wealth according to the results of the transactions in which it was committed. 4. each city updates its population according to its new resulting wealth

Design concepts
Emergence : The desired phenomonon to emerge is the diversity of cities trajectories (observed at the whole system scale) and the upkeep of the hierarchisation in the population distribution. Hierarchy itself is pre-existent as it is introduced via the census data of 1959.
Adaptation : Adapatation occurs when the outcomes of transactions betwen cities (gain or loss of wealth) are reported on cities' population (stage 8 of process overview).

Objectives, Learning, Prediction
: In the simple model presented in this paper, there is no cognitive dimension in the behaviour of cities.
Sensing : Hypotheses are made that each city is able to know the demand, the supply and the localisation of every other city in the system.
Interactions : The only interactions between cities in this simple model are the confrontation of supplies and demand, resulting in transactions. This confrontation is framed by a gravity model , defining a asymmetric matrix of interaction potential based on supplies and demands of cities, and distances separating them.

Stochasticity, Collectives
: There is no stochasticity nor collectives considerations in this generic model.
Observation : Basically, the whole population of the system is observed at each time step. The main focus of observation is set on the population distribution, over time, for the 1145 cities of the system (for a comparison purpose with empirical data, to obtain calibration error) . We also monitor synthetic variable distributions (wealth, supply and demand), especially to highlight ruined cities or over-producing/ over-consuming cities.
Initialization : Population of each city is initialized with the population registered in the census of 1959.
Input Data : In this version, of the model does not use input data to represent time-varying processes.

Sub-Models :
Here are the details of the four points of the process overview. Equations are numbered like in the paper.
• each city updates its economic variables according to its population (supply, demand).
The wealth attribute of each city is computed via th relation of Equation 2, resulting in a superlinear distribution of wealth.

Eq.2 :
Supplies and Demands are computed likewise, via a scaling exponent expressing the effect of the size of the city on its productivity and consumption per inhabitant.

Eq.3 :
Eq.4 : • each city interacts with the others according to the intensity of their interaction potential. For two distinct cities, A and B, an interaction consists in confronting A's supply to B's demand, resulting in a transaction of goods sent from A to B.
Interaction potential between two cities i and j is computed following Equation 6 Eq.6 :

Interurban exchange of values
A city i attributes a share of its supply S ij to a city j that is proportional to the share of the interaction potential F ij in the total interaction potential of the city i :

Eq.7 :
Symetrically, a city i attributes a share of its demand D ij to a city j that is proportional to the share of the interaction potential F ji in the total interaction potential of the city i : Eq.8 : The effective transaction from a city i to a city j is determined as : Eq. 9 : • each city updates its wealth according to the results of the transactions in which it was committed.
Wealth is updated acording Equation 10 Eq.10 : • each city updates its population according to its new resulting wealth The variation of pupulation is computed following Equation 11 Eq.11 : Then the population is updated Eq.12 :