The idea behind simulation models is that we can use them to make predictions of what might happen in the real world. As we want often want to use the results of these simulations to inform decision making it is important for the results to be accurate and precise. The way we do this is by keeping a careful eye on the errors.
Traditionally people have focused on trying to minimise the error that crops up in the actual simulation itself, the technical name for this is the ‘simulation-estimation error‘.
However there is another important error that is often overlooked and needs to be taken into account as well, the ‘input-uncertainty error‘. This is to do with the uncertainty in the actual values you plug into your simulation to start with.
Consider a classic M/M/ queuing system with an arrival rate and a mean service time . If we don’t know what the true values of and actually are, then we have to estimate them from what we observe in the real world.
If we observe m arrivals into the system noting their inter-arrival times and their service times , then we can estimate the true values of and by,
Imagine that we then run replications (repeats) of our queuing system simulation and observe the number of customers in the system once we have reached steady state as .
The true number of people in the system once we have reached steady state is estimated as . Then clever people have found estimates for the expectation and variance of the approximated steady state mean to be,
The term is the input-uncertainty that we are interested in. We can see the relative importance of this error as compared to the simulation-estimation error . If we make n large then the simulation-estimation error will decrease, but at some point it will be smaller than the input-uncertainty error.
Since the input-uncertainty error can dominate the simulation-estimation error, it is clear to see that we need to know about the input-uncertainty error. This is particularly true for extremely precise problems.
As you might expect real problems are not as simple as this example. We can often have lots of inputs into our simulation which complicates matters and the output is often a highly non-linear (complicated!) function of the inputs.
This means that we can’t write down the output and explicitly see the what the input-uncertainty error is. So we need new methods for quantifying input-uncertainty such as bootstrapping and confidence intervals.