Information criteria are statistical measures that are employed in selecting the best model from a set of competing models. These statistical measures take into account both the goodness of fit of the model and its complexity, ensuring that the selected model is not overfit or underfit.
Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are two popular information criteria used for model selection.
AIC is based on the principle that a good model should minimize the information loss during the prediction process. It measures the quality of a model through the difference between the maximum likelihood of the model and the number of model parameters. AIC is formulated as follows:
AIC = − 2log(L) + 2k
where L is the likelihood of the model and k is the number of parameters in the model.
Unlike AIC, BIC takes into account the complexity of the model when computing the likelihood. It assumes that the true model is among the set of candidate models and assigns a higher penalty to models with more parameters. BIC is formulated as follows:
BIC = − 2log(L) + log(n)k
where n is the sample size and the other terms are the same as in AIC.
The main advantage of using information criteria in model selection is that they balance the trade-off between model complexity and goodness of fit. They provide a principled way of selecting the most parsimonious model that best captures the patterns in the data.
For example, consider a linear regression problem with multiple predictors. Suppose we have two models - one with three predictors and another with six predictors. To select the best model, we can compute the AIC and BIC scores for each model and choose the one with the lower value. The model with 3 predictors is likely to have a lesser AIC and BIC value, indicating that it may be the better choice as compared to the model with 6 predictors. This approach ensures that the selected model is not overfit and generalizes well to unseen data.