Model comparison

Model evidence

In the context of Bayesian statistics, the most rigorous way of comparing a set of models is looking at the Bayes evidence associated with each model. Given a dataset \(\mathcal D\), model \(\mathcal M\) and a set of parameter \(\theta\), the evidence or marginal likelihood is given by

\[\mathcal{Z} = P(\mathcal D|\mathcal M) = \int_\theta P(\mathcal D | \theta, \mathcal M) P(\theta | \mathcal M) \mathrm{d}\theta = \int_\theta \mathcal{L}_{\mathcal M}(\theta) \pi(\theta) \mathrm{d}\theta\]

To compare different models, one wants to compute the Bayes factor, which is given by

\[R = \frac{P(\mathcal{M}_1 | d)}{P(\mathcal{M}_2 | d)} = \frac{P(\mathcal{D} | \mathcal{M}_1) P(\mathcal{M}_1)}{P(\mathcal{D} | \mathcal{M}_2) P(\mathcal{M}_2)} = \frac{\mathcal{Z}_1}{\mathcal{Z}_2} \times \frac{P(\mathcal{M}_1)}{P(\mathcal{M}_2)} .\]

If there is no a priori reason for preferring one model over the other (as it is often the case), the prior ratio \(P(\mathcal{M}_1)/P(\mathcal{M}_2)\) becomes unity and the Bayes factor becomes the ratio of evidences.

Information criteria