Validation and Calibration
The first thing would be to define terms - validation and calibration. Validation refers to methods for judging how accurate a model is in predicting the outputs. The extent of this accuracy is referred to as validity, of which there are five types
face validity: the extent to which the problem formulation and model structure correspond to current evidence as subjectively judged by clinical experts or other stakeholders with expertise in the condition.
internal validity: the extent to which the model equations and code are accurate as well as the consistency of those calculations with the model structure
cross-validity: the extent to which the model produces similar results to other models intended for the same purpose
predictive validity: the extent to which the model predicts results for data not yet observed
external validity: the extent to which the model’s results corresponding to real-world data
Calibration is a set of methods for improving external validity, commonly implemented by adjusting unobserved or unavailable1 model parameters to plausible values so that the model’s output matches real-world data. For example, in the MiMiC-BlaKy model, one of the calibrated parameters was the annual probability of diagnosis for each stage of undiagnosed cancer. Others were the annual probability of moving from a low-risk bladder to a high-risk bladder cancer progression pathway, speed of tumour progression through stages in undiagnosed patients, and a yearly reduction in the probability of symptomatic diagnosis for individuals over the age of 75 years. These parameters cannot be observed, as data on cancer relates to diagnosed patients only.
1 Parameters are unobserved when it is not possible to record these, such as the progression of undiagnosed cancer. Unavailable parameters, however, can in principle be observed but not are missing, owing to data limitations
Model Calibration Methods
Once the calibration parameters have been selected, the steps in calibration include identifying the calibration target, selecting the goodness of fit (GOF) measure, choosing the parameter search algorithm, specifying the acceptance criteria, and specifying a stopping rule.