In standard machine learning, the starting point typically consists of empirical loss functions such as Mean Squared Error or Cross-Entropy. Later, probabilistic ML introduces concepts like priors, likelihoods, and posteriors, which can make the field appear fragmented.
In reality, these approaches are mathematically equivalent. The relationship becomes clear when structured into three progressive levels: Maximum Likelihood (MLE), Maximum A Posteriori (MAP), and full Bayesian inference.
1. The Core Mechanism: Updating Beliefs
Bayes’ theorem formalizes how an initial belief regarding a parameter or hypothesis $\theta$ is updated once dataset $\mathcal{D}$ has been observed:
$$ \underbrace{p(\theta \mid \mathcal{D})}_{\text{Posterior}} = \frac{\overbrace{p(\mathcal{D} \mid \theta)}^{\text{Likelihood}} \cdot \overbrace{p(\theta)}^{\text{Prior}}}{\underbrace{p(\mathcal{D})}_{\text{Evidence / Marginal Likelihood}}} $$- Prior $p(\theta)$: The initial assumption about $\theta$ (such as model weights) before observing data.
- Likelihood $p(\mathcal{D} \mid \theta)$: The probability of the observed data given a specific parameter configuration $\theta$.
- Posterior $p(\theta \mid \mathcal{D})$: The updated probability distribution over $\theta$ after incorporating the evidence.
- Evidence $p(\mathcal{D}) = \int p(\mathcal{D} \mid \theta) p(\theta) d\theta$: A normalizing constant ensuring that the posterior integrates to 1.
Because the evidence $p(\mathcal{D})$ does not depend on $\theta$, it acts purely as a normalizer during parameter optimization:
$$ \text{Posterior} \propto \text{Likelihood} \cdot \text{Prior} $$The updated belief is therefore a direct compromise between empirical evidence (Likelihood) and prior constraints (Prior).
Sequential Updating
Bayesian updating is naturally recursive. When data arrives sequentially in separate subsets $\mathcal{D}_1$ and $\mathcal{D}_2$, the posterior obtained from the first batch acts directly as the prior for the second:
$$ p(\theta \mid \mathcal{D}_1, \mathcal{D}_2) \propto p(\mathcal{D}_2 \mid \theta) \, p(\theta \mid \mathcal{D}_1) $$Rather than retraining the model on the entire combined history from scratch, parameter distributions can be continually refined as new observations stream in.
2. Why Priors Matter: The Base Rate Fallacy
To see why relying solely on likelihood can lead to misleading conclusions, consider a diagnostic medical test:
- Prior: Only 1 in 1,000 individuals has a specific disease, so $p(\text{Sick}) = 0.001$.
- Likelihood (Sensitivity): The test correctly identifies the disease 99% of the time, so $p(\text{Positive} \mid \text{Sick}) = 0.99$.
- False Positive Rate: The test incorrectly flags 5% of healthy individuals, so $p(\text{Positive} \mid \text{Healthy}) = 0.05$.
If a positive test result is obtained, the probability that the individual is actually sick evaluates to:
$$ p(\text{Sick} \mid \text{Positive}) = \frac{0.99 \cdot 0.001}{(0.99 \cdot 0.001) + (0.05 \cdot 0.999)} \approx \frac{0.00099}{0.00099 + 0.04995} \approx 1.94\% $$Even with a 99% accurate test, the true probability of having the disease remains below 2%. Because the healthy population is vastly larger than the infected population, false alarms significantly outnumber true positives. The prior anchors the estimate and prevents rare events from being drastically overestimated.
Level 1: Maximum Likelihood (Standard Loss Functions)
In probabilistic machine learning, model parameters $\theta$ are often treated as unknown but fixed constants, without placing a prior distribution over them.
In regression setups, a model $f_\theta(x)$ is used to predict continuous labels $y$. Under the assumption of additive zero-mean Gaussian observation noise:
$$ y_i = f_\theta(x_i) + \varepsilon_i, \quad \varepsilon_i \sim \mathcal{N}(0, \sigma^2) $$Assuming the samples in dataset $\mathcal{D} = {(x_i, y_i)}_{i=1}^N$ are independent and identically distributed (i.i.d.) conditioned on $\theta$, the joint likelihood factorizes into a product across all observations:
$$ p(\mathcal{D} \mid \theta) = \prod_{i=1}^N p(y_i \mid x_i, \theta) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(y_i - f_\theta(x_i))^2}{2\sigma^2} \right) $$For computational stability, the objective is evaluated in log-space, turning the product into a sum. Since optimization algorithms are designed for minimization, the negative log-likelihood (NLL) is used:
$$ -\log p(\mathcal{D} \mid \theta) = \frac{1}{2\sigma^2} \sum_{i=1}^N (y_i - f_\theta(x_i))^2 + \underbrace{\frac{N}{2}\log(2\pi\sigma^2)}_{\text{const}} $$Notice: Because the second term does not depend on $\theta$, its gradient vanishes with respect to the parameters:
$$\frac{\partial}{\partial \theta} \left( \frac{N}{2}\log(2\pi\sigma^2) \right) = 0$$
Minimizing the negative log-likelihood with respect to $\theta$ yields:
$$ \hat{\theta}_{\text{MLE}} = \arg\min_\theta \sum_{i=1}^N (y_i - f_\theta(x_i))^2 $$This is mathematically identical to Mean Squared Error (MSE). Minimizing MSE is therefore equivalent to Maximum Likelihood Estimation under the assumption of independent, normally distributed errors.
If Laplace noise were assumed instead, the negative log-likelihood would yield the $L_1$ absolute error loss. If binary targets with Bernoulli noise were used, it would yield Binary Cross-Entropy.
Level 2: Maximum A Posteriori (Priors as Regularization)
Maximum Likelihood optimizes purely for fit on the training data, which leaves models vulnerable to overfitting when sample sizes are small.
When a prior distribution $p(\theta)$ is assigned to the parameters, the single parameter vector that maximizes the posterior density can be computed. This defines the Maximum A Posteriori (MAP) estimate:
$$ \hat{\theta}_{\text{MAP}} = \arg\max_\theta p(\theta \mid \mathcal{D}) = \arg\max_\theta \left[ \log p(\mathcal{D} \mid \theta) + \log p(\theta) \right] $$Reformulated as an equivalent minimization problem:
$$ \hat{\theta}_{\text{MAP}} = \arg\min_\theta \Big[ \underbrace{-\log p(\mathcal{D} \mid \theta)}_{\text{Data Loss}} + \underbrace{\left( -\log p(\theta) \right)}_{\text{Regularizer}} \Big] $$Deriving Weight Decay from a Gaussian Prior
An isotropic Gaussian prior is assumed, meaning all $P$ parameter dimensions are independent and identically distributed (i.i.d.) with zero mean and variance $\sigma_0^2$, written as $\theta \sim \mathcal{N}(0, \sigma_0^2 I)$. Due to independence, the prior density factorizes across dimensions:
$$ p(\theta) = \prod_{j=1}^P p(\theta_j) = \prod_{j=1}^P \frac{1}{\sqrt{2\pi\sigma_0^2}} \exp\left( -\frac{\theta_j^2}{2\sigma_0^2} \right) $$The negative log-prior evaluates to:
$$ -\log p(\theta) = \frac{1}{2\sigma_0^2} \underbrace{\sum_{j=1}^P \theta_j^2}_{\|\theta\|_2^2} + \underbrace{\frac{P}{2}\log(2\pi\sigma_0^2)}_{\text{const}} $$Substituting both the Gaussian likelihood and the Gaussian prior into the MAP objective yields:
$$ \hat{\theta}_{\text{MAP}} = \arg\min_\theta \left[ \frac{1}{2\sigma^2} \sum_{i=1}^N (y_i - f_\theta(x_i))^2 + \frac{1}{2\sigma_0^2} \|\theta\|_2^2 \right] $$Multiplying the entire objective by $\frac{2\sigma^2}{N}$ preserves the minimum:
$$ \hat{\theta}_{\text{MAP}} = \arg\min_\theta \left[ \operatorname{MSE}(\theta) + \lambda \|\theta\|_2^2 \right], \quad \text{where } \lambda = \frac{\sigma^2}{N \sigma_0^2} $$This directly recovers $L_2$ regularization (Ridge regression / Weight Decay):
- A Gaussian prior on the parameters corresponds to an $L_2$ penalty.
- A Laplace prior (sharply peaked at zero) corresponds to an $L_1$ penalty (Lasso), driving weights to exact zeros.
- The regularizer weight $\lambda$ represents the ratio between noise variance $\sigma^2$ and prior belief variance $\sigma_0^2$, scaled inversely by sample size $N$.
Level 3: Full Bayesian Inference
Although MAP estimation incorporates Bayes’ theorem, it still produces a single point estimate $\hat{\theta}_{\text{MAP}}$. Once optimization finishes, all information about parameter uncertainty is lost.
In contrast, full Bayesian inference avoids selecting a single parameter configuration and preserves the complete posterior distribution $p(\theta \mid \mathcal{D})$.
Predictions for an unseen data point $x^*$ are obtained by integrating across all possible parameter settings, weighted by their posterior probability:
$$ p(y^* \mid x^*, \mathcal{D}) = \int p(y^* \mid x^*, \theta) \, p(\theta \mid \mathcal{D}) \, d\theta $$This is the posterior predictive distribution. Predictions do not rely on a single parameter setting, but represent an expectation over the entire parameter space.
Why MAP is More Common in Deep Learning
Analytical evaluation of this integral is feasible only for tractable cases (such as linear models with conjugate priors). For non-linear architectures like deep neural networks, exact computation is intractable because the marginal likelihood $p(\mathcal{D})$ cannot be calculated in closed form.
Consequently, full Bayesian inference requires approximate methods:
- MCMC (Markov Chain Monte Carlo): Generates parameter samples drawn directly from the posterior.
- Variational Inference (VI): Approximates the true posterior with a simpler distribution family (such as a factorized Gaussian) via numerical optimization.
- Monte Carlo Dropout / Deep Ensembles: Practical approximations that sample diverse network weights to estimate predictive uncertainty.
Summary
| Approach | Objective | Parameter Representation | Practical ML Equivalent |
|---|---|---|---|
| MLE | $\arg\max_\theta \log p(\mathcal{D} \mid \theta)$ | Single point estimate | Unregularized Loss (MSE, Cross-Entropy) |
| MAP | $\arg\max_\theta [\log p(\mathcal{D} \mid \theta) + \log p(\theta)]$ | Single point estimate | Regularized Loss ($L_2$ Weight Decay, $L_1$ Lasso) |
| Full Bayes | Compute $p(\theta \mid \mathcal{D})$ | Full probability distribution | Ensemble / Bayesian Predictive Distribution |