revisiting statistical learning: maximum likelihood and Bayesian estimation

Recently, while revisiting statistical machine learning, I found that many concepts had become blurry. I no longer remembered some derivations and principles, so I decided to organize them again, starting with the two major statistical schools: frequentist and Bayesian.

Maximum likelihood estimation (MLE)

The basic idea of maximum likelihood is to find the model parameters that make the observed data most probable.

Principle

1. Assume data comes from probability distribution $P(x|\theta)$, where $\theta$ is the parameter to estimate.
2. Given observations $\{x_1, x_2, \ldots, x_n\}$, define the likelihood as $L(\theta) = P(x_1, x_2, \ldots, x_n | \theta)$.
3. Assuming the data is independent and identically distributed (i.i.d.), the joint probability becomes $L(\theta) = \prod_{i=1}^n P(x_i|\theta)$.
4. Taking logs simplifies the calculation: $\ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log P(x_i|\theta)$.
5. Find the parameter that maximizes log likelihood: $\theta_{MLE} = \arg\max_{\theta} \ell(\theta)$.

MLE represents the traditional frequentist position: it relies only on data and no prior distribution. With enough samples, its expectation equals the true parameter. With insufficient samples or too many parameters, it can overfit.

Bayesian estimation

Bayesian estimation combines prior knowledge and observed data through Bayes’ theorem to obtain a posterior distribution over parameters.

$$P(\theta | x) = \frac{P(x | \theta) P(\theta)}{P(x)}.$$

• $P(\theta)$ is the prior distribution, representing knowledge of $\theta$ before observing data.
• $P(x|\theta)$ is the likelihood of observing $x$ under $\theta$.
• $P(x)$ is the normalizing constant, which is usually not directly important.
• $P(\theta|x)$ is the posterior distribution after combining the prior and observations.

The posterior can be used in two common ways:

• Maximum a posteriori (MAP): $\theta_{MAP} = \arg\max_{\theta} P(\theta | x)$.
• Posterior mean: ${\theta}_{Bayes} = \int \theta P(\theta|x)\,d\theta$.

Because it incorporates prior information, Bayesian estimation can perform better than MLE when samples are scarce.

Differences between MLE and Bayesian estimation

Feature	Maximum likelihood estimation	Bayesian estimation
Theoretical basis	Frequentist; focuses on observed data	Bayesian; combines prior and data
Input	Observed data only	Observed data and prior distribution
Objective	Maximize likelihood	Maximize posterior
Suitable setting	Large samples; no reliable prior	Small samples; reliable prior available
Computational cost	Lower; analytic solutions are often possible	Higher; may need numerical integration or approximation
Result	Point estimate	Posterior distribution or point estimate

A more concrete explanation

Consider tuning a machine-learning model. MLE treats the model parameter as a fixed value and finds the value that maximizes data likelihood. Bayesian estimation treats it as random: it begins with a prior distribution, then uses the data to calculate a posterior.

1. In MLE, maximize the log-likelihood directly to find the parameter.
2. In Bayesian estimation, suppose weights are centered at zero and normally distributed. That becomes the prior: $P(\theta) = \prod_{j=1}^d N(\theta_j; 0, \tau^2)$.
3. Bayes’ rule then yields a posterior proportional to the product of the prior and likelihood. Maximizing it gives:

$$\theta_{\text{MAP}} = \arg\max_{\theta} \left( \log P(y \mid x; \theta) + \log P(\theta) \right).$$

Compared with MLE, MAP differs only by the prior term $\log P(\theta)$.

A personal question: do previous batches become priors?

I wondered whether, when training with multiple batches, each batch’s model carries prior knowledge from the previous batch, including its model and data.

The answer is no for MLE. MLE has no prior-probability concept. Parameters are fixed but unknown, and the goal is to maximize likelihood. In mini-batch training, each batch contributes locally to the full-data log likelihood:

$$\ell(\theta) = \sum_{i=1}^n \log P(x_i | \theta).$$

Each batch changes the current parameters to improve their fit to observed data, but it does not change any prior assumption about $\theta$. The model state after one batch is an optimization state, not a dynamically updated prior.

Bayesian estimation explicitly distinguishes prior knowledge from data. After observing a batch, its posterior is:

$$P(\theta | x_{batch}) \propto P(x_{batch}|\theta)P(\theta).$$

That posterior can become the prior for the next batch. This sequential posterior-to-prior update is a defining Bayesian property.

An analogy: MLE mini-batch training is like adding numbers in pieces to calculate one total; every batch contributes to the total likelihood. Bayesian learning is a step-by-step belief update, where each batch changes the belief used as the starting point for the next batch.

Empirical risk and structural risk

Introductory statistical learning distinguishes empirical and structural risk. In MLE, empirical risk is the likelihood-based loss minimized on observed data. Structural risk is the regularization term. When a model is a conditional probability distribution, its loss is log loss, and model complexity is represented by the prior probability, maximizing the posterior is structural risk minimization—exactly the MAP formula.

January 24, 2025, Suzhou