In an early draft of Jonathan Huggins’ and Jeff Miller’s BayesBag paper I learned of a particular ``paradox’’ in Bayesian model selection, in which different models with precisely equal explanatory power are not asymptotically given equal posterior probability by a Bayesian posterior. Rather, the Bayes posterior concentrates entirely on one model or the other, though on which model it concetrates is determined by a fair coin flip. In other words, the symmetry between the two equally good models is maintained on resampling the data, but the Bayesian posterior uncertainty doesn’t seem to adequately represent this symmetry for any particular dataset. Apparently the idea goes back to a 1966 paper by Berk (see ibid. for the full citation), so I’ve been calling it Berk’s paradox, though I’m not sure that’s standard. I found that the following simple decomposition of a log likelihood ratio helped me to understand Berk’s paradox, and some other issues besides.
Let’s introduce some notation. Suppose we independent and identically distributed data, , and that is large enough for standard asymptotic approximations to hold. Say we have two different probabilistic models for models, parameterized by and . In a standard abuse of notation, we’ll write the probabilty of under these two models as and , respectively. Identities that hold for either model will be written in terms of . Denote log probabilities by . Finally, suppose that the true distribution of the data is given by (which is not necessary either of the models), and denote the KL divergence from to by .
Write the true optimal parameters as
and write the maximum likelihood estimators as
We will be interested in the posterior probability that is the data generating process, which we denote . For simplicity, take the equiprobable prior . By Bayes’ rule,
Let us now do a simple asymptotic analysis of . For reasonable priors , by the Laplace approximation as ,
We can decompose the key quantity into three terms, each of a different asymptotic order:
Let us give these terms their own descriptive names:
so that . As I now show, in general we expect these terms to be , , and , respectively.
The first term, , is the average difference between the log likelihood at the empirical optima and at the population optima . By an analysis similar to the proof of Wilke’s theorem, one can show that .
The second term, , is by a standard central limit theorem argument unless the variance is zero.
The final term, , is times the relative KL divergence of the two models, evaluated at the “true” optimal parameters, since
Consequently, unless the two models have equal divergence to the truth.
From this decomposition, a number of interest facts follow immediately.
A consistency result is clearest. If the model is a better fit in terms of KL divergence, then , and at rate , dominating the other terms. Consequently, . Conversely, if the model is a better fit, then and . This consistency result obtains despite the randomness in the data, which is of smaller order.
The second consequence is Berk’s paradox. Suppose that the two models are equally explanatory so that and, consequently, . In that case, the term dominates. Since , with equal prbability goes to or according to whether is greater or less than zero, respectively. Correspondingly, goes to either or zero respectively with equal probability.
In finite samples, Berk’s paradox can occur with different severity depending on the variance . Specifically, if and are highly correlated, then will be small, and will approach its extremes relatively slowly. This occurs when the two models are very similar to one another — imagine, for example, comparing two very similar specific parameter values of a single parametric model.
Note also that, in finite samples, Berk’s paradox can occur even if the two models are not exactly equivalent if the KL difference is of order smaller than .
Third, one can see clearly why the frequentsist likelihood ratio test comes with the restriction that the tested models be nested. Under the null hypothesis that the two models are equivalent to one another, almost surely, so , and is zero. Of course, the null equivalence of the two models implies that is zero as well. When, additionally, the two models are equivalent to , then the term has a distribution by classical results (Wilke’s theorem), and it is this distribution that is used to test the null.
Were one to apply the frequentist likelihood ratio test with equivalent but non-nested models, then would not be zero, and one would also suffer from the phenomenon of Berk’s paradox. In particular, one would asymptotically reject the null exactly half the time for any finite cutoff.
Practical Consequences of Berk’s paradox
Is Berk’s paradox simply an ivory tower peculiarity? I do not think it is. On the contrary, I believe it illustrates a fundamental brittleness of Bayesian pairwise model selection, particularly when we need Bayesian thinking most—when there is real model uncertainty, i.e., when is not near or .
Suppose you’re a workaday Bayesian, and you want to perform a pairwise model comparison using Bayes’ rule. As you are instructed by your textbooks, you calculate . Suppose your models are complicated so you have no idea what the posterior’s frequentist properties are. Recall that you don’t actually observe the terms , , and because you don’t know , , or .
What if is close to one? You don’t know whether this is the case because or has dominated. In other words, you don’t really know whether the models are nearly equivalent and the high confidence posterior is due to chance, or because the posterior is extremely confident due to chance. Still, you can take some solace from the consistency result that even small differences in divergence dominate asymptotically.
What if is at some value that is not close to nor to ? Then you really don’t know much except that the divergence is probably small. Either is very small, and the models have nearly the same likelihood (in which case comparing them is uninteresting), or is not small and you would, with all likelihood, not come to the same conclusion with a new draw from the same generating distribution.
Sparse regression example
Here’s a concrete example of something I, at least, would have done wrong before thinking about Berk’s paradox in this way. Suppose you have a sparse linear model and can actually calculate or estimate the posterior probability of different sparsity patterns. Denote a sparsity pattern $M_s$, for $s$ ranging over the power set of including or excluding regressors. Before I was aware of Berk’s paradox, I would have happily taken all patterns such that was sufficintly close to or and considered those regressors to be confidently included or excluded. Then I would have taken all the others, for which was at some intermediate value, and treated them as “uncertain” for further, more careful analysis. Unfortunately, as show above, this procedure will be extremely brittle to resampling the data. One will not pick up the same pattern of “certain” and “uncertain” regressors with new datasets drawn from the same generating distribution, even if the data actually comes from one of the specified models. (One can verify with simulations that this phenomenon does in fact occur with sparse linear regression.)
As I write this, I’m not sure what the solution is. A careful reading of the proof of Theorem 4.1 in the BayesBag paper shows that they are using the bootstrap to estimate and then rescale the test statistic. I think that is a good idea, though I do wonder whether the BayesBag posterior is more than is needed to effect this solution.
I should emphasize that the above objections only apply to the attempt to choose one model over another, or over a group of alternative models. Bayesian model averaging (BMA) does not really suffer from this problem, in that it does not matter for the purpose of BMA which of several nearly equivalent models you choose.
So I currently feel a bit lost. If you are reading this and have ideas or thoughts, please do reach out!