ELBO

In Variational Bayesian Methods, the Evidence Lower Bound (ELBO) is a lower bound on the log-likelihood of some observed data.

Terminology and Notation

Let $X$ and $Z$ be random variables, jointly distributed with distribution $p_\theta$. For example, $p_\theta(X)$ is the Marginal Distribution of $X$, and $$p_\theta(Z

X)$is the conditional distribution of$Z$given$X$. There, for any samle$x \sim p_\theta$, and any distribution$q_\phi$$, we have

\[\ln{p_\theta}(x) \geq \mathbb{E}_{z\sim q_\phi}\left[\ln{\frac{p_\theta(x, z)}{q_\phi(z)}}\right].\]

LHS: evidence for $x$ RHS: evidence lower bound (ELBO) for $x$ The above is refered as the ELBO inequality.

Applying

To derive the ELBO, we introduce Jensen’s Inequality applied to randam variables $x \in X$ here:

\[\begin{align} f(\mathbb{E}[X]) \leq \mathbb{E}[f(X)] \end{align}\]

We apply Jensen’s Inequality to the $\log$ (marginal) probability of the observations to get the ELBO.

\[\begin{align} \log p(x) &= \log\int_z{p(x, z)dz} \\ &= \log\int_z{p(x, z)\frac{q(z)}{q(z)}dz} \\ &= \log\int_z{\frac{p(x, z)}{q(z)}q(z)dz} \\ &= \log\left({\mathbb{E}_{q(z)}\left[ {\frac{p(x, z)}{q(z)}}\right]}\right) \\ &\geq \mathbb{E}_{q(z)}\left[ \log{\frac{p(x, z)}{q(z)}} \right] \\ &= \mathbb{E}_{q(z)}\left[ \log{p(x, z)} \right] - \mathbb{E}_{q(z)}[\log{q(z)}] \end{align}\]

All together, the ELBO for a probability model $p(x, z)$ and an approximation $q(z)$ to the posterior is: $\mathbb{E}_{q(z)}[\log{p(x, z)}]-\mathbb{E}_{q(z)}[\log{q(z)}]$