Modern machine learning is incredible at finding statistical patterns in large datasets. Neural networks regularly reach or beat human performance in vision, speech, and language tasks.

However, relying only on statistical correlations comes with a major catch: standard models assume that the test data follows the exact same distribution as the training data. In the real world, data distributions shift. When this occurs, purely statistical models often fail.

To build models that are robust to distribution shifts, we have to look beyond pure correlation and turn to causal inference.

1. Statistical vs. Causal Dependence

The fundamental rule of causal inference is simple: association does not imply causation. Mathematically, observing a variable is not necessarily the same as actively setting it:

$$ p(Y \mid X) \neq p(Y \mid do(X)) $$

A simple example is the correlation between beachside ice cream sales ($X$) and drowning accidents ($Y$). A standard supervised model trained on historical data will use ice cream sales to predict drowning risk.

This statistical link exists because both variables share a common cause: warm weather ($Z$). Hot temperatures increase both ice cream sales and the number of peole swimming at the beach:

$$ X \leftarrow Z \rightarrow Y $$

If an authority steps in and bans ice cream sales entirely, the number of drownings will not change at all. The statistical correlation disappears under intervention because ice cream sales is not cause of the drownings.

The Problem with Empirical Risk Minimization

Standard supervised learning relies on Empirical Risk Minimization (ERM):

$$ \hat{\theta} = \arg\min_\theta \frac{1}{N} \sum_{i=1}^N \mathcal{L}(f_\theta(x_i), y_i) $$

Gradient descent is greedy: It grabs any correlation in the training data that reduces the loss. This leads to shortcut learning:

  • Classifying cows based on green grass backgrounds instead of animal features.
  • Detecting pneumonia in chest X-rays based on hospital-specific metal markers instead of lung tissue changes.

Standard regularization (like $L_1$ or $L_2$ weight decay) does not solve this. Regularization keeps weights small to prevent the model from fitting random noise, but it cannot tell whether a strong association is a genuine cause or just a spurious correlation. If a background shortcut reliably predicts the label across the training split, regularized models will still rely on it.

Causal Graphs

To prevent models from relying on shortcuts, a formal way to describe cause and effect is needed. In causal inference, this structure is represented as a Directed Acyclic Graph (DAG):

  • Each variable is a node in the graph.
  • A directed arrow from $A$ to $B$ ($A \rightarrow B$) means that $A$ is a direct physical cause of $B$.
  • The graph is acyclic (contains no directed loops) because an effect cannot travel backwards to cause its own cause.

This graphical view changes how the joint probability distribution of all variables $(X_1, \dots, X_d)$ is broken down.

In standard probability theory, the chain rule allows factorizing a joint distribution in any arbitrary order. For example, $p(A, B)$ can be written as $p(B \mid A) p(A)$ or as $p(A \mid B) p(B)$. Purely statistically, both factorizations are identical.

In the real world, however, only the factorization that follows the direction of the causal arrows reflects how the data was actually generated. In a causal DAG, the system is decomposed so that each variable conditions exclusively on its direct causal parents $\text{pa}(X_j)$:

$$ p(X_1, \dots, X_d) = \prod_{j=1}^d p(X_j \mid \text{pa}(X_j)) $$

This decomposition is the causal factorization. Looking at the graph, every term in the product represents an isolated node that listens only to its direct incoming arrows.

Independent Causal Mechanisms

This modular structure leads directly to the Independent Causal Mechanisms (ICM) principle. The physical world is modeled like a collection of autonomous functions: each node computes its value based only on its direct parents, without knowing or caring how the other variables were computed.

This modularity explains why causal representations remain robust under distribution shifts:

  • When an intervention takes place or the environment changes, only that specific local mechanism $p(X_j \mid \text{pa}(X_j))$ is modified.
  • All other mechanisms and their conditional probabilities remain invariant.

Standard neural networks fail out-of-distribution, because they ignore this modularity. By exploiting non-causal shortcuts (such as predicting a cause from an effect or relying on confounders), they entangle multiple independent mechanisms into a single statistical relationship. As soon as one part of the environment shifts, that entangled relationship collapses.

2. The Ladder of Causation

Judea Pearl structures reasoning into three distinct levels:

LevelCapabilityCore QuestionFormal Expression
1. AssociationObservingWhat does seeing $X$ tell us about $Y$?$p(Y \mid X)$
2. InterventionActingWhat happens to $Y$ if we set $X$?$p(Y \mid do(X))$
3. CounterfactualsRetrospectingWhat would $Y$ have been if $X$ had been different, given what actually happened?$p(Y_{X=0} \mid X=1, Y=1)$

Standard machine learning lives entirely on Level 1. It models passive conditional probabilities without knowing whether an association is causal.

Level 2: Interventions and Graph Surgery

Level 2 introduces the $do$-operator. The distribution $p(Y \mid do(X = x))$ models what happens to $Y$ when variable $X$ is actively fixed to value $x$.

Graphically, the operation $do(X = x)$ corresponds to graph surgery: we cut all incoming arrows pointing into $X$.

Cutting the incoming edges isolates X from its usual cause Z. This breaks the confounding path: any remaining association between X and Y must now be purely causal (which, in the case of ice cream and drownings, is zero).

Level 3: Counterfactuals and SCMs

Level 3 asks what would have happened in a specific situation if things had gone differently. Because we cannot observe two parallel realities for the same individual, graph manipulation alone is not enough.

Counterfactuals require a Structural Causal Model (SCM). In an SCM, each variable is determined by an assignment function and an unobserved noise term $U$:

$$ X_j := f_j(\text{pa}(X_j), U_j) $$

The assignment operator $:=$ represents an asymmetric physical rule. Changing the right side updates $X_j$, but changing $X_j$ does not alter its causes.

To answer a counterfactual, we first use the observed data to infer the hidden noise values $U$, replace the targeted function with our hypothetical value, and then recalculate the outcome.

3. Advantages of Causal Machine Learning

Two key areas in modern machine learning benefit directly from adopting a causal perspective: robustness and interpretability.

1. Robustness Against Distribution Shifts

Standard models greedily use any correlation that minimizes training loss, making them fragile when test data differs from training conditions. Causal approaches aim to find a representation $\Phi(X)$ that captures only invariant causal factors:

$$ p(Y \mid \Phi(X)) \quad \text{remains invariant across all environments } e \in \mathcal{E} $$

Frameworks like Invariant Risk Minimization (IRM) use training data collected from multiple environments (such as different hospitals or lighting conditions). Since spurious correlations change between settings, the model learns to ignore unstable shortcuts and keeps only the stable causal relationships.

2. Interpretability and Actionable Recourse

Popular explainability tools (like saliency maps or SHAP) are associative: they show which features statistically correlated with an output, not what would happen if a feature were actively changed.

In sensitive domains like healthcare or credit approval, users need actionable guidance rather than correlation metrics:

  • Associative explanation: “Your loan was rejected because your debt score correlates with defaults.”
  • Actionable recourse: “Decreasing existing debt by amount $d$ ($do(\text{Debt} = d)$) will change the decision to approved.”

A causal approach ensures that recommended explanations reflect genuine cause and effect, preventing advice that relies on non-causal side effects.

4. Challenges

Combining causality with machine learning is an active research area. While there are still big theoretical challenges, there is a lot of recent progress:

1. Causal Representation Learning

  • The Problem: Standard causal inference requires clear conceptual variables (like medical test results or age). Deep learning, however, deals with raw pixels or text tokens. Furthermore, trying to extract true causal concepts purely from unsupervised data is mathematically impossible without extra assumptions (an issue known as non-identifiability).
  • Current Directions:
    • Using Multi-Environment Data: Instead of training on one static dataset, models learn from data collected across different environments, time steps, or small interventions (e.g. i-VAE or CITRIS). Knowing that an environment shifted gives the model just enough guidance to isolate true latent factors.
    • Object-Centric Architectures: Models like Slot Attention build in architectural biases that encourage neural networks to group visual scenes into distinct, movable objects rather than treating an image as a flat grid of pixels.

2. Causal Discovery (Learning the Graph)

  • The Problem: In most real-world problems, nobody provides the true causal graph. Finding directed edges from observational data alone is tricky because multiple different graphs can produce the exact same correlations. On top of that, checking every possible graph combination is computationally impossible for more than a few variables.
  • Current Directions:

    • Differentiable Discovery (NOTEARS): Instead of searching through graphs one by one with discrete algorithms, modern methods turn graph structure into a continuous adjacency matrix. This allows gradient descent to iteratively optimize the graph’s edges.

    • Exploiting Asymmetries: If relationships in the data are non-linear with additive noise ($Y = f(X) + \text{noise}$), the statistical footprint looks different in the causal direction than in the reverse direction. Algorithms can use this asymmetry to decide which way an arrow should point without needing active experiments.

3. Feedback Loops and Dynamic Systems

  • The Problem: Causal DAGs do not allow directed cycles. However, real-world systems (like financial markets, ecosystems, or biology) are full of feedback loops over time: $X$ affects $Y$, which then affects $X$ a moment later.
  • Current Directions:
    • Unrolling Over Time: In time-series data, each time step is treated as its own node ($X_t \rightarrow Y_{t+1} \rightarrow X_{t+2}$). Because causes always happen before effects in time, the unrolled graph remains strictly acyclic.
    • Combining SCMs with Differential Equations: Researchers use tools like Neural ODEs to model systems continuously over time. In this view, a static causal graph is simply the resting equilibrium of an underlying dynamic system.

4. Computational Cost

  • The Problem: Answering an intervention question ($do$-query) or calculating counterfactuals usually requires summing or integrating over hidden confounders or latent vectors. For large neural networks, this integration becomes intractable (i.e. too slow for practical use).
  • Current Directions:
    • Causal VAEs: Instead of running expensive integration at test time, an encoder network is trained to approximate the distribution directly in a single forward pass.
    • Double Machine Learning (DML): This approach uses standard machine learning models (like Random Forests or Neural Networks) to isolate and control for background confounders. This makes it possible to estimate clean causal effects quickly, even on massive datasets.

Literature

Foundations and Intuition

  • Judea Pearl, Dana Mackenzie (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.
    Overview
    (A non-technical introduction to the Ladder of Causation and causal reasoning.)
  • Jonas Peters, Dominik Janzing, Bernhard Schölkopf (2017). Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press
    (The standard textbook for causal factorization, ICM, and additive noise models.)

Why Classical ML Fails

  • Robert Geirhos et al. (2020). Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence.
    arXiv:2004.07780
    (Explores how standard empirical loss minimization leads models to rely on spurious correlations.)
  • Martin Arjovsky et al. (2019). Invariant Risk Minimization.
    arXiv:1907.02893
    (The foundational paper on learning invariant causal representations across environments.)

Current Frontiers

  • Bernhard Schölkopf et al. (2021). Toward Causal Representation Learning. IEEE Proceedings.
    arXiv:2102.11107
    (The main roadmap for bridging low-level sensory data with high-level causal variables.)
  • Xun Zheng et al. (2018). DAGs with NO TEARS: Continuous Optimization for Structure Learning. NeurIPS.
    arXiv:1803.01422
    (Introduces differentiable causal discovery by turning acyclicity into an algebraic equality constraint.)
  • Victor Chernozhukov et al. (2018). Double/Debiased Machine Learning for Treatment and Structural Parameters. The Econometrics Journal.
    arXiv:1608.00060
    (The foundational framework for estimating causal effects efficiently using standard ML models.)