Validation of Simulation Models: Regression Analysis Revisited
33 Pages Posted: 26 Nov 1996
Date Written: Undated
Abstract
This paper proves that it is wrong to require that regressing a model's outputs on the observed real outcomes give a 45 line through the origin (unit slope, zero intercept). Therefore this paper proposes an alter-native requirement: the responses of the model and the real system should have the same means and the same variances. To test whether this requirement is satisfied, a novel statistical procedure is derived. This procedure regresses the differences of simulated and real responses on their sums. The old and the new procedures are investigated in an extensive Monte Carlo experiment that simulates queueing systems. The conclusions of this experiment are that (i) the old test rejects a valid simulation model substantially more often than the novel test does; (ii) the intuitive test shows 'perverse' behavior in a certain domain: the worse the simulation model, the higher its probability of acceptance; and (iii) the novel test does not reject a valid simulation model too often (its type I error probability is correct), provided the queueing response is transformed logarithmically.
JEL Classification: C10, C15
Suggested Citation: Suggested Citation
Do you have negative results from your research you’d like to share?
Recommended Papers
-
Constrained Optimization in Simulation: A Novel Approach
By Jack P. C. Kleijnen, Wim C. M. Van Beers, ...
-
Robust Optimization in Simulation: Taguchi and Response Surface Methodology
By Gabriella Dellino, Jack P. C. Kleijnen, ...
-
By Lonneke Driessen, R.c.m. Brekelmans, ...
-
Methodology for Determining the Acceptability of Given Designs in Uncertain Environments
By Jack P. C. Kleijnen, Henri Pierreval, ...
-
White Noise Assumptions Revisited: Regression Models and Statistical Designs for Simulation Practice
-
Robust Optimization in Simulation: Taguchi and Krige Combined
By Gabriella Dellino, Jack P. C. Kleijnen, ...
-
Expected Improvement in Efficient Global Optimization Through Bootstrapped Kriging
By Jack P. C. Kleijnen, Wim C. M. Van Beers, ...