Bayesian information criterion

In statistics, the Bayesian information criterion (BIC) or Schwarz information criterion (also SIC, SBC, SBIC) is a criterion for model selection among a finite set of models; models with lower BIC are generally preferred. It is based, in part, on the likelihood function and it is closely related to the Akaike information criterion (AIC).

When fitting models, it is possible to increase the maximum likelihood by adding parameters, but doing so may result in overfitting. Both BIC and AIC attempt to resolve this problem by introducing a penalty term for the number of parameters in the model; the penalty term is larger in BIC than in AIC for sample sizes greater than 7.^[1]

The BIC was developed by Gideon E. Schwarz and published in a 1978 paper,^[2] as a large-sample approximation to the Bayes factor.

Definition

The BIC is formally defined as^[3]^[a]

\mathrm {BIC} =k\ln(n)-2\ln({\widehat {L}}).\

where

${\hat {L}}$ = the maximized value of the likelihood function of the model $M$ , i.e. ${\hat {L}}=p(x\mid {\widehat {\theta }},M)$ , where { ${\widehat {\theta }}$ } are the parameter values that maximize the likelihood function and $x$ is the observed data;
$n$ = the number of data points in $x$ , the number of observations, or equivalently, the sample size;
$k$ = the number of parameters estimated by the model. For example, in multiple linear regression, the estimated parameters are the intercept, the $q$ slope parameters, and the constant variance of the errors; thus, $k=q+2$ .

Derivation

The BIC can be derived by integrating out the parameters of the model using Laplace's method, starting with the following model evidence:^[5]^[6]^: 217

p(x\mid M)=\int p(x\mid \theta ,M)\pi (\theta \mid M)\,d\theta

where $\pi (\theta \mid M)$ is the prior for $\theta$ under model $M$ .

The log-likelihood, $\ln(p(x|\theta ,M))$ , is then expanded to a second order Taylor series about the MLE, ${\widehat {\theta }}$ , assuming it is twice differentiable as follows:

\ln(p(x\mid \theta ,M))=\ln({\widehat {L}})-{\frac {n}{2}}(\theta -{\widehat {\theta }})^{\operatorname {T} }{\mathcal {I}}(\theta )(\theta -{\widehat {\theta }})+R(x,\theta ),

where ${\mathcal {I}}(\theta )$ is the average observed information per observation, and $R(x,\theta )$ denotes the residual term. To the extent that $R(x,\theta )$ is negligible and $\pi (\theta \mid M)$ is relatively linear near ${\widehat {\theta }}$ , we can integrate out $\theta$ to get the following:

p(x\mid M)\approx {\hat {L}}{\left({\frac {2\pi }{n}}\right)}^{\frac {k}{2}}|{\mathcal {I}}({\widehat {\theta }})|^{-{\frac {1}{2}}}\pi ({\widehat {\theta }})

As $n$ increases, we can ignore $|{\mathcal {I}}({\widehat {\theta }})|$ and $\pi ({\widehat {\theta }})$ as they are $O(1)$ . Thus,

p(x\mid M)=\exp \left(\ln {\widehat {L}}-{\frac {k}{2}}\ln(n)+O(1)\right)=\exp \left(-{\frac {\mathrm {BIC} }{2}}+O(1)\right),

where BIC is defined as above, and ${\widehat {L}}$ either (a) is the Bayesian posterior mode or (b) uses the MLE and the prior $\pi (\theta \mid M)$ has nonzero slope at the MLE. Then the posterior

p(M\mid x)\propto p(x\mid M)p(M)\approx \exp \left(-{\frac {\mathrm {BIC} }{2}}\right)p(M)

Usage

When picking from several models, ones with lower BIC values are generally preferred. The BIC is an increasing function of the error variance $\sigma _{e}^{2}$ and an increasing function of k. That is, unexplained variation in the dependent variable and the number of explanatory variables increase the value of BIC. However, a lower BIC does not necessarily indicate one model is better than another. Because it involves approximations, the BIC is merely a heuristic. In particular, differences in BIC should never be treated like transformed Bayes factors.

It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable^[b] are identical for all models being compared. The models being compared need not be nested, unlike the case when models are being compared using an F-test or a likelihood ratio test.^{[citation needed]}

Properties

The BIC generally penalizes free parameters more strongly than the Akaike information criterion, though it depends on the size of n and relative magnitude of n and k.
It is independent of the prior.
It can measure the efficiency of the parameterized model in terms of predicting the data.
It penalizes the complexity of the model where complexity refers to the number of parameters in the model.
It is approximately equal to the minimum description length criterion but with negative sign.
It can be used to choose the number of clusters according to the intrinsic complexity present in a particular dataset.
It is closely related to other penalized likelihood criteria such as Deviance information criterion and the Akaike information criterion.

Limitations

The BIC suffers from two main limitations^[7]

the above approximation is only valid for sample size $n$ much larger than the number $k$ of parameters in the model.
the BIC cannot handle complex collections of models as in the variable selection (or feature selection) problem in high-dimension.^[7]

Gaussian special case

Under the assumption that the model errors or disturbances are independent and identically distributed according to a normal distribution and the boundary condition that the derivative of the log likelihood with respect to the true variance is zero, this becomes (up to an additive constant, which depends only on n and not on the model):^[8]

\mathrm {BIC} =n\ln({\widehat {\sigma _{e}^{2}}})+k\ln(n)\

where ${\widehat {\sigma _{e}^{2}}}$ is the error variance. The error variance in this case is defined as

{\widehat {\sigma _{e}^{2}}}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-{\widehat {x_{i}}})^{2}.

which is a biased estimator for the true variance.

In terms of the residual sum of squares (RSS) the BIC is

\mathrm {BIC} =n\ln(RSS/n)+k\ln(n)\

When testing multiple linear models against a saturated model, the BIC can be rewritten in terms of the deviance $\chi ^{2}$ as:^[9]

\mathrm {BIC} =\chi ^{2}+k\ln(n)

where $k$ is the number of model parameters in the test.

Notes

^ The AIC, AICc and BIC defined by Claeskens and Hjort^[4] are the negatives of those defined in this article and in most other standard references.
^ A dependent variable is also called a response variable or an outcome variable. See Regression analysis.

References

^ See the review paper: Stoica, P.; Selen, Y. (2004), "Model-order selection: a review of information criterion rules", IEEE Signal Processing Magazine (July): 36–47, doi:10.1109/MSP.2004.1311138, S2CID 17338979.
^ Schwarz, Gideon E. (1978), "Estimating the dimension of a model", Annals of Statistics, 6 (2): 461–464, doi:10.1214/aos/1176344136, MR 0468014.
^ Wit, Ernst; Edwin van den Heuvel; Jan-Willem Romeyn (2012). "'All models are wrong...': an introduction to model uncertainty" (PDF). Statistica Neerlandica. 66 (3): 217–236. doi:10.1111/j.1467-9574.2012.00530.x. S2CID 7793470.
^ Claeskens, G.; Hjort, N. L. (2008), Model Selection and Model Averaging, Cambridge University Press
^ Raftery, A.E. (1995). "Bayesian model selection in social research". Sociological Methodology. 25: 111–196. doi:10.2307/271063. JSTOR 271063.
^ Konishi, Sadanori; Kitagawa, Genshiro (2008). Information criteria and statistical modeling. Springer. ISBN 978-0-387-71886-6.
^ ^a ^b Giraud, C. (2015). Introduction to high-dimensional statistics. Chapman & Hall/CRC. ISBN 9781482237948.
^ Priestley, M.B. (1981). Spectral Analysis and Time Series. Academic Press. ISBN 978-0-12-564922-3. (p. 375).
^ Kass, Robert E.; Raftery, Adrian E. (1995), "Bayes Factors", Journal of the American Statistical Association, 90 (430): 773–795, doi:10.2307/2291091, ISSN 0162-1459, JSTOR 2291091.

External links

Sparse Vector Autoregressive Modeling

[5] The AIC, AICc and BIC defined by Claeskens and Hjort^[4] are the negatives of those defined in this article and in most other standard references.

[8] A dependent variable is also called a response variable or an outcome variable. See Regression analysis.

[1] See the review paper: Stoica, P.; Selen, Y. (2004), "Model-order selection: a review of information criterion rules", IEEE Signal Processing Magazine (July): 36–47, doi:10.1109/MSP.2004.1311138, S2CID 17338979.

[2] Schwarz, Gideon E. (1978), "Estimating the dimension of a model", Annals of Statistics, 6 (2): 461–464, doi:10.1214/aos/1176344136, MR 0468014.

[3] Wit, Ernst; Edwin van den Heuvel; Jan-Willem Romeyn (2012). "'All models are wrong...': an introduction to model uncertainty" (PDF). Statistica Neerlandica. 66 (3): 217–236. doi:10.1111/j.1467-9574.2012.00530.x. S2CID 7793470.

[4] Claeskens, G.; Hjort, N. L. (2008), Model Selection and Model Averaging, Cambridge University Press

[6] Raftery, A.E. (1995). "Bayesian model selection in social research". Sociological Methodology. 25: 111–196. doi:10.2307/271063. JSTOR 271063.

[7] Konishi, Sadanori; Kitagawa, Genshiro (2008). Information criteria and statistical modeling. Springer. ISBN 978-0-387-71886-6.

[Giraud-9] Giraud, C. (2015). Introduction to high-dimensional statistics. Chapman & Hall/CRC. ISBN 9781482237948.

[Priestley-10] Priestley, M.B. (1981). Spectral Analysis and Time Series. Academic Press. ISBN 978-0-12-564922-3. (p. 375).

[Raftery1995-11] Kass, Robert E.; Raftery, Adrian E. (1995), "Bayes Factors", Journal of the American Statistical Association, 90 (430): 773–795, doi:10.2307/2291091, ISSN 0162-1459, JSTOR 2291091.

[1]

[2]

[3]

[a]

[5]

[6]

[b]

[7]

[8]

[9]

[4]