|
The accuracy of a simulation model is measured by the closeness of the model output to that of the real systems. Since a number of assumptions about the behaviour of real systems are made in developing the model, there are two steps in measuring the accuracy. The first step is checking whether or not the assumptions are reasonable. The second step is whether or not the model implements those assumptions correctly. These two steps are called validation and verification, respectively. Validation is concerned with the representativeness of the assumptions, and verification is related to the correctness of the implementation. Verification can also be called debugging, that is, ensuring that the model does what it is intended to do. In the programming language literature, a number of techniques can be found for debugging. Any combination of these techniques can be used to verify the model.
I.5.2 Model Validation Techniques
Validation refers to ensuring that the assumptions used in developing the model are reasonable in that, if correctly implemented, the model would produce results close to that observed in real systems. The validation techniques depend upon the assumptions and, hence, on the system being modelled. Thus, unlike verification techniques that are generally applicable, the validation techniques used in one simulation may not apply to another. One of the classical papers in the validation literature is that of Naylor and Finger (1967), where a three-step approach is given for validating a simulation model:
1. Build a model that has high face validity. 2. Validate model assumptions. 3. Compare the model input-output t ransformations to corresponding input-output transformation for the real system
The primary objective during the first step is to develop a model with high face validity, i.e., a model that, on the surface, seems reasonable to people who are knowledgeable about the system under study. In order to develop such a model, the simulation modellers should make use of all existing information, including the following:
conversations with system "experts"; observations of the system; existing theory; relevant results from similar simulation models; experience/intuition.
Test the Assumptions of the Model Empirically. The goal of the second step of validation is to test quantitatively the assumptions made during the initial stages of model development. One of the most useful tools during the second step of validation is sensitivity analysis. This can be used to determine if the simulation output changes significantly when the value of an input parameter is changed, when an input probability distribution is changed, or when the level of detail for a subsystem is changed. If the output is sensitive to some aspect of the model, then that aspect must be modelled carefully. Determine How Representative the Simulation Output Data Are The most definitive test for a simulation model's validity is establishing that its output data closely resembles the output data that would be expected from the actual (proposed) system. If a system similar to the proposed one exists, then a simulation model of the existing system is developed and its output data are compared to those from the existing system itself. If the two sets of data compare "favourably", then the model of the existing system is considered "valid". The model is then modified so that it represents the proposed system. The greater the communality between the existing and proposed systems, the greater our confidence in the model of the proposed system. There is no completely definitive approach for validating the model of the proposed system. If there were, there might be no need for a simulation model in the first place. If there is no an existing system similar to the proposed system or if there is an existing system but no definitive output data, then it is still worthwhile to have system experts review the simulation output data for reasonableness. (Care must be taken in performing this exercise, since if one knew exactly what output to expect, there would be no need for a model.) Animation may also be an effective way for experts to evaluate the validity of a simulation model. Up to now we have discussed validating a simulation model relative to past or present system output data; however, a perhaps more definitive test of a model is to establish its ability to predict future system behaviour, when the model input data match the real inputs and when a policy implemented in the model is implemented at some point in the system. In other words, the structure of the model should be accurate enough for the model to make good predictions, not just for one input data set, but for the range of input data sets, which are of interest.
I.5.3 Transient Removal
In most simulations, only the steady-state performance, that is, the performance after the system has reached a stable state, is of interest. In such cases, results of the initial part of the simulation should not be included in the final computations. This initial part is also called the transient state. The problem of identifying the end of the transient state is called transient removal. The main difficulty with transient removal is that it is not possible to define exactly what constitutes the transient state and when that transient state ends. All methods for transient state removal are therefore heuristic. Examples of such methods are: (i) simply use very long runs; that is, runs that are long enough to ensure that the presence of initial conditions will not affect the result; or (ii) use proper initialisation, which requires starting the simulation in a state close to the expected steady state; and (iii) various other methods that are based on the assumption that the variability during the steady state is less than during the transient state, which is generally true.
I.5.4 Stopping Criteria: Variance Estimation
It is important that the length of the simulation be properly chosen. If the simulation is too short, the results may be highly variable. On the other hand, if the simulation is too long, computing resources and manpower may be unnecessarily wasted. The simulation should run until the confidence interval for the mean response narrows to a desired width.
|
|