Do Not Trust Too Short Sequential Simulation

Dyna, 2009

Recibido para revisar julio 25 de 2008, aceptado octubre 28 de 2008, versión final noviembre 20 de 2008 RESUMEN: Los estudios de simulación, empleados en diversas áreas de la investigación, son de gran utilidad para conocer el comportamiento de ciertos fenómenos bajo diferentes escenarios virtuales propiciados por el investigador a través de algún software especializado. En el campo de la estadística son muy comunes los estudios de robustez, muchos de ellos utilizados para observar el comportamiento de un estimador ante diferentes situaciones hipotéticas que pudieran presentarse en la realidad. Dada la semejanza entre los estudios de simulación y los estudios experimentales, el objetivo de este trabajo es proponer el uso de la metodología y del diseño y análisis de experimentos en los estudios de simulación. Se presenta como ejemplo el estudio de simulación "Robustez de un modelo de Markov de tres estados bajo distintas especificaciones distribucionales de los tiempos de transición" realizado bajo el enfoque del diseño experimental y utilizando el análisis de varianza y la regresión múltiple con el fin de probar el efecto del tamaño muestral, el máximo número de visitas y el tipo de distribución de los tiempos de transición sobre la estimación de los parámetros del modelo markoviano de tres estados.

2004

ABSTRACT The method of Spectral Analysis proposed by Heidelberger and Welch (SA/HW) is an effective and efficient way of calculating the error of a sample mean in sequential simulation. A simple modification to the method improves the coverage of the resulting estimators in the case of sequential simulation.

Simulation Practice and Theory, 1998

Stochastic discrete-event simulation has become one of the most-used tools for performance evaluation in science and engineering. But no innovation can replace the responsibility of simulators for obtaining credible results from their simulation experiments. In this paper we address the problem of the statistical correctness of simulation output data analysis, in the context of sequential steady-state stochastic simulation, conducted for studying long run behaviour of stable systems. Such simulations are stopped as soon as the relative precision of estimates, defined as the relative half-width of confidence intervals at a specified confidence level, reaches the required level. We formulate basic rules for the proper experimental analysis of the coverage of steady-state interval estimators. Our main argument is that such an analysis should be done sequentially. The numerical results of our coverage analysis of the method of Non-overlapping Batch Means and Spectral Analysis are presented, and compared with those obtained by the traditional, non-sequential approach. Two scenarios for stochastic simulation are considered: traditional sequential simulation on a single processor, and fast concurrent simulation based on Multiple Replications in Parallel (MRIP), with multiple processors cooperating in the production of output data.

1999

ABSTRACT It is generally accepted that sequential stochastic simulation is the only practical approach allowing control of the statistical error of the results of steady-state stochastic simulation. A commonly used stopping rule of such stochastic simulation is based on the relative statistical error, since then the magnitude of the point estimates does not need to be known beforehand.

I would like to thank my husband, Vaughn, and children Keagan and Ryan. Thank you for enduring my long hours, and family separation so I could pursue this goal. Special thanks to Dr. Joyce A. Nagle and Dr. Matthew S. Gerber for your time in reviewing my work and providing valuable feedback. Thanks to Yue Zang for working with me on the Repast simulation, Matlab coding and the many discussions we had on the police staffing study. Thank you to my committee for sticking with me through this process over the past several years. Your patience, willingness, and guidance were very much appreciated. I would also like to thank my advisor, Professor Donald E. Brown for giving me direction. You were always available to look over my work and provide feedback whatever the hour. Finally, I must thank Constance Brown for giving me encouragement and a cup of tea when I needed both.

One of the most important but neglected aspects of a simulation study is the proper design and analysis of simulation experiments. In this tutorial we give a state-of-the-art presentation of what the practitioner really needs to know to be successful. We will discuss how to choose the simulation run length, the warmup-period duration (if any), and the required number of model replications (each using different random numbers). The talk concludes with a discussion of three critical pitfalls in simulation output-data analysis.

International Journal of Agent Technologies and Systems

This article discusses the question of whether significance tests on simulation results are meaningful at all. It is also argued that it is the effect size much more than the existence of the effect is what matters. It is the description of the distribution function of the stochastic process incorporated in the simulation model which is important. This is particularly when this distribution is far from normal, which is particularly often the case when the simulation model is nonlinear. To this end, this article uses three different agent-based models to demonstrate that the effects of input parameters on output metrics can often be made “statistically significant” on any desired level by increasing the number of runs, even for negligible effect sizes. The examples are also used to give hints as to how many runs are necessary to estimate effect sizes and how the input parameters determine output metrics.

Simulation Modelling Practice and Theory

In discrete event steady-state simulation, deletion the initial transient phase of the simulation is usually recommended in order to reduce. Various heuristics and tests have been proposed to aid with this. The plummeting cost of simulation, combined with uncertainties about the overall reliability of transient methods, suggest revisiting the notion that deletion is essential, especially for longer simulations. We consider this in a sequential simulation framework. The results show, that for run lengths giving commonly used relative precisions of the results, there is no substantial difference in point or interval estimates of means due to deleting the initial transient. However in sequential simulation, deleting the initial transient turns out to have considerable value in reducing the chance of premature stopping, thus ensuring that the accuracy of the final results is closer to that specified.

1995

The contents of this work reflect the views of the au thors who are responsible for the facts and accuracy of the data presented. Responsibility for the application of the material to specific cases, however, lies with any user of the report and no responsibility in such cases will be attributed to the author or to the University of Canter bury. This technical report contains a research paper, develop ment report or tutorial article which has been submitted for publication in a journal or for consideration by the comissioning organization. We ask you to respect the current and future owner of the copyright by keeping copying of this article to the essential minimum. Any requests for further copies should be sent to the author.

2013

Computer simulations facilitate the study of real world phenomena by providing safe, controlled, flexible, and repeatable experimental environments at costs far lower than other options. Both the validity of the simulation model and the scope of its validity determine the degree to which the findings can be generalized and applied to real world situations and problems. In practice, no single model captures all of the important aspects of a phenomena of interest, nor is applicable over a wide set of missions and circumstances. Thus, effectively utilizing a variety of models in a prospective meta-analysis (a set of common hypotheses and controllable variables and comparable metrics) offers the opportunity to improve validity and extend the findings to a broader range of real world situations. NATO SAS-085, a research group exploring C2 Agility and Requisite Maturity, conceived and developed an international meta-analysis approach for studying various aspects related to C2 Agility from...