Postdoctoral Research Associate
Simulation games are models that replicate dynamic problems in the form of an interactive simulator for the purpose of training or analysis. Examples of commercially available games can be found online: Strategy Dynamics, Forio, ABSEL, MIT. Although the simulation games developed nowadays are almost exclusively computer-based, earlier examples include board games, like the famous Beer Distribution Game.
Simulation games have many advantages, as explained by Sterman (1994): They provide low-cost laboratories for learning. The virtual world allows time and space to be compressed or dilated. Actions can be repeated under the same or different conditions. One can stop the action to reflect. Decisions that are dangerous, infeasible, or unethical in the real system can be taken in the virtual world. One can try strategies that one suspects will lead to poor performance or even (simulated) catastrophe. Virtual worlds provide high- quality outcome feedback. The degree of random variation in the virtual world can be controlled. Virtual worlds offer the learner greater control over strategy.
Simulation games are helpful in both learning and research purposes. They are used as tools for identifying information, heuristics and mental models used by subjects as well as effects of internal and external factors on the performance. The researchers use the simulation games as a simple controlled environment in which they can test their hypothesis on human psychology and behavior. They try to give answer questions like: “Do people perform better in a familiar context?”, “Do people ignore nonlinearites in dynamic decision making?”, “How practice affects performance?”. One of the potential outcome of this type of research is that we can hypothesize and test how simulation games may “be improved to facilitate further research as well as learning” (Davidsen, 2000). The simulation games are also used in practice to help the decision makers to develop a mental model that will make them perform better in similar real problems.
Ask any academic or professional, they will tell that real-life is complex. Whether it is the global warming problem, the traffic problem in a city, the mortgage crisis, or the task of adjusting water temperature in the shower, we -humans- have difficulties in understanding, analyzing and managing complex systems. These systems are not complex only because of the sheer size of the problem, but because the variables involved change over time, they interact with other variables, there are delays, nonlinear relations and feedback loops. Delays separate the cause and effect in time, and thus make it difficult to relate a consequence to previous actions causing it. Nonlinearity corresponds to a situation where the relationship between the cause and its effect is not linear. It is a result of saturation effects, limited resources and interactions. Note that almost all real-life systems are nonlinear. It is difficult to control a nonlinear system because the outputs will not be the proportional to the inputs, and we usually do not know the shape of nonlinear relation. Feedback corresponds to a situation where two variables affect each other through a chain of circular causal relationships. Feedback is expected to bring a complexity because it makes the system harder to predict as our actions are reinforced or balanced out of our control. The complexity as a result of such factors is called 'systemic complexity'.
Although simulation games are simplifications of real problems, they still have inherent systemic complexity due to factors such as accumulation, delay, feedback and nonlinearities. The effects of systemic complexity factors have been analyzed using simulation games in the literature. For example, studies show that (Booth Sweeney & Sterman, 2000; Cronin et al., 2009) even educated people may be unable to infer the behavior of a system in the presence of a simple stock. Various studies have analyzed the relationship between delay and game performance (Broadbent & Aston, 1978; Diehl, 1989; Sterman, 1989b; Paich & Sterman, 1993; Diehl & Sterman, 1995; Barlas & Ö̈zevin, 2004) and many report a negative effect of delay on performance. It is also seen from research that, strength of feedback can be effective on the game performance (Diehl, 1989; Kampmann, 1992; Paich & Sterman, 1993; Diehl & Sterman, 1995; Langley et al., 1998). Likewise, it is known that nonlinearity can deteriorate the performance (Sterman, 1989a; Sterman, 1989b; Paich & Sterman, 1993). In spite of this wide range studies, most papers in the literature focus on one complexity factor at a time, with a few noteworthy exceptions. Paich & Sterman (1993) uses a boom-and-bust type of game to test the effects of feedback strength and delay on game performance. The paper tests two levels of each factor in a two-by-two design. Gary & Wood (2005) extends this work by including more cognition-related variables and by using mental model accuracy as another performance measure. Diehl & Sterman (1995) uses a stock management game to tests the effects on feedback strength and delay using more levels, based on A three-by-five Latin square design.
My thesis analyzes the effects of multiple complexity factors with many levels on a given same game, to reach conclusions about the dynamic complexity of a simulation game. This research is particularly distinct from earlier work in the sense that; (1) Four different dynamic complexity factors are tested: delay duration, delay order, nonlinearity and feedback, each at many levels, (2) the interactions between factors are analyzed as well as the main effects, (3) a stock management game as well as a growth game is used, which is relatively uncommon compared to vast literature of stock-management games, (4) both game scores and players’ subjective difficulty assessments are used as complexity measures, (5) rigorous analytical study is carried out to interpret the results from a dynamic complexity perspective.
To analyze the effects of complexity factors, a two-stage experimental design is followed. In the first stage, the factors are analyzed at many levels, without interaction. Then, based on the results of the first stage, we select one level for each factor, and analyze their interaction effects. To increase generality of the conclusions, we use two types of simulation games: a growth management game that requires the player to sustainably increase the profit using price and advertising; and a stock management game in which the player has to bring the inventory level to a target, with minimum deviations. These two games are not only different in terms of cover stories and structures, but also in terms of how complexity factors affect the game performances.
The aim of the first stage experiments was determining levels for each complexity factor where they start to make the game significantly complex with respect to a very simple base game, which does not include any systemic complexity factor. This seemingly straightforward step turned out be more complicated and interesting than expected. We discovered that the complexity factors do not necessarily deteriorate the game performance. In the growth management game, although the average performance in games involving delay is worse than the base game, the direction of delay duration’s effect is opposite of the expected: the worst performance is observed when delay is shortest and as it gets longer the performance improves. We discovered that this unexpected result is due to a side effect of delay that can temporarily facilitate growth in a finite time horizon. Similar unexpected results are also observed for nonlinearity and feedback. The performances in games involving all levels of nonlinearity are superior to the base game performances. Also, as feedback (that suppresses growth) gets stronger, the performance is strangely improved, so that best performance scores are observed when feedback is strongest. We carried out further analysis to understand the source of these counter-intuitive results. We concluded that combined effect of learning and structural changes brought by the nonlinear effect function improves the scores of nonlinearity games. Likewise, feedback is shown to increase performance relative to a benchmark since its suppressing effect also shows its effect on the benchmark. In the stock management game, on the other hand, the results were closer to the expected. Delay has a deteriorating effect when its order is high and duration is long. The effect of delay duration is also observed in the subjective difficulty ratings. Despite trying extreme shapes of nonlinear function, nonlinearity is not found to be effective on the game performance. But, subjective difficulty ratings indicate a slight increase in perceived difficulty with increasing level of nonlinearity. Similarly, feedback turned to be insignificant.
Given the fact that the above first stage experiments did not turn out to be as expected, the task determining a level for each complexity factor for the second stage became non-trivial. For the factors that improved performance in the first stage, we set them to levels where they start to create an improvement in game performance, to see if the interaction effects would be in the same or opposite direction. The second stage experiments show that interaction is indeed an important component in systemic complexity. In the growth management game all factors in isolation turned out to be improving performance, as expected. The interesting conclusion of the experiment is that the effects of complexity factors may be amplified or suppressed when they are in interaction with each other. Statistical analysis indicate that delay–feedback interaction improves the performance, but interaction of nonlinearity with other two factors deteriorate the scores. In the stock management game, while delay is the only factor that deteriorates the performance by itself, delay–feedback also turned out to be significant.
In addition to experimental analysis regarding complexity and learning, my thesis also presents findings on heuristic rules specifically designed for the growth and stock management games. A method is presented for obtaining statistical distributions of scores resulting from simulations of such heuristics. The results show that under some complexity conditions, human subjects do not perform better than the random heuristic —a primitive rule composed of a sequence of random decisions. The developed non-random heuristics on the other hand, yield performances that mimic subject behaviors reasonably well.
The complexity-orinted experiments yield interesting results in terms of learning, as well. In the first stage experiments of the growth management game, only the players playing games involving nonlinearity show indications of procedural learning by repetition. In addition to evidence of limited learning, there are also signs of “false” learning: the scores of the final base game of the delay group is not different from the delay games’ scores. In the stock management game, all groups exhibit performance improvement with repeated trials. The results change when interaction comes into the picture. In the second stage experiments, the effect of repeated trials is not significant for neither of the game types.
The final part of the thesis directly focuses on learning. In this stage, we propose a procedure in which games are played in an increasing order of complexity, and test its effectiveness on game performance, conceptual learning and transfer of learning. Using controlled experiments, we test whether playing simpler versions of a game in an increasing complexity order helps learning and performing in a complex game, as opposed to playing the simpler versions in random order, or repeatedly playing the same complex game without increasing the order of complexity.
2014 By Onur Özgün