Maximum Likelihood Estimation (MLE)

A maximum likelihood estimation (MLE) is a method of estimating the parameters of the given likelihood probability distribution.

Definition

A value of \(\theta\) that maximizes \(L(\theta|x_1, x_2, ..., x_n)\). Most likely, natural log will be plugged (Log-Likelihood Function).

\[\begin{align*} \theta^* &= \text{argmax}_\theta l(\theta|x_1, x_2, ..., x_n) \\ &= \text{argmax}_\theta log(\mathcal{L}(\theta|x_1, x_2, ..., x_n)) \\ &= \text{argmax}_\theta log(\prod_{i=1}^{n}f(x_i|\theta)) \\ &= \text{argmax}_\theta log(f(x_1|\theta) \times f(x_2|\theta) \times ... \times f(x_n|\theta)) \\ &= \text{argmax}_\theta \sum_{i=1}^{n}log(f(x_i|\theta)) \end{align*}\]

From Bayesian Backpropagation:

\[\begin{align*} w^\text{MLE} &= \text{argmax}_wl(w|\mathcal{D}) \\ &= \text{argmax}_w\log{\mathcal{L}(w|\mathcal{D})} \\ &= \text{argmax}_w\log{P(\mathcal{D}|w)} \\ &= \text{argmax}_w\log{P(\mathcal{D_1}, ...\mathcal{D_n}|w)} \\ &= \text{argmax}_w\log{\prod_{i=1}^{n}P(\mathcal{D_i}|w)} \\ &= \text{argmax}_w\sum_{i=1}^{n}{\log{P(\mathcal{D_i}|w)}} \\ &= \text{argmax}_w\sum_{i=1}^{n}{\log{P(y_i|x_i,w)}} \\ \end{align*}\]