Beyond the Average Treatment Effect: Who Actually Benefits?

Much of applied causal inference begins with a deceptively simple question: does the treatment work?

The average treatment effect gives us one answer.

But an average can hide economically important heterogeneity. The same intervention may generate large effects for some people, small effects for others, and almost no effect for another group.

From ATE to CATE

Conditional average treatment effects allow treatment effects to vary with observed characteristics X:

This moves the analysis from a single population average toward systematic treatment-effect heterogeneity.

But there is an important distinction: discovering that an outcome is predictable across groups is not the same as establishing that the causal effect of treatment differs across those groups.

A Practical Example: 401(k) Eligibility and Wealth

I built a companion empirical project in Python using a public U.S. 401(k) dataset to study how eligibility for a 401(k) plan is related to household net financial assets.

The dataset contains 9,915 households. The treatment is 401(k) eligibility, the outcome is net financial assets, and the adjustment set includes demographic and economic characteristics such as age, income, education, family size, marital status, dual-earner status, pension characteristics, IRA participation, and homeownership.

The raw difference is striking.

But this is an observed association, not automatically a causal effect. Households eligible for 401(k) plans may differ systematically from households that are not eligible.

Adjustment Changes the Story

After adjusting for observed covariates, several estimators produce substantially smaller effects than the raw difference.

The contrast is economically meaningful. A raw difference of almost $19,600 falls substantially once observed differences between eligible and noneligible households are taken into account.

A causal interpretation still depends on the identifying assumptions—not simply on the estimator.

When the Average Hides the Economics

An average effect around $8,000 is useful, but it does not tell us whether the effect is approximately $8,000 for everyone.

To investigate heterogeneity, I estimated conditional treatment effects using Causal Forest DML.

The estimated CATE distribution is right-skewed. The mean estimated CATE is approximately $8,056, while the median is approximately $5,864.

That difference already suggests that a single average does not fully describe the estimated treatment-effect distribution.

Heterogeneity Across Income

One of the clearest empirical patterns appears when estimated treatment effects are summarized by income quintile.

The pattern is not perfectly monotonic at the bottom of the income distribution, but estimated effects become much larger among higher-income households.

This should not be interpreted as evidence that income itself causes greater responsiveness. Income may summarize many correlated dimensions of household financial circumstances.

Heterogeneity Across Age

Estimated effects also differ across age groups.

Again, these estimates describe conditional heterogeneity under the maintained causal assumptions. They are not observed individual treatment effects.

Why Use a Causal Forest?

Traditional subgroup analysis requires the researcher to specify the dimensions of heterogeneity in advance.

Causal forests provide a flexible way to estimate how treatment effects vary with a potentially richer set of covariates while allowing nonlinearities and interactions.

Double machine learning and cross-fitting help separate nuisance estimation from treatment-effect estimation and reduce the risk that flexible prediction models simply overfit the same observations used to estimate the causal parameter.

But the flexibility of machine learning does not solve the fundamental identification problem.

A Different Learner, a Similar Pattern

As a robustness exercise, I also estimated heterogeneous effects using a T-learner.

The exact estimates differ, but the broad income gradient remains similar across the two approaches.

Agreement across estimators is useful as a robustness check, but it is not proof of causal identification. Two sophisticated estimators can still share the same identifying assumptions.

Causal Machine Learning Does Not Eliminate Identification

Flexible algorithms can model complex relationships in the data, but they cannot determine whether treatment assignment is conditionally independent of potential outcomes.

In an observational setting such as this one, causal interpretation requires assumptions such as conditional exchangeability and positivity.

The overlap analysis in this project shows approximately 99.4% common support, which is encouraging for estimation. But overlap is necessary, not sufficient, for identification.

From “Does It Work?” to “For Whom?”

The broader workflow is:

Prediction asks what is likely to happen. Causal inference asks what would change if we intervened. Causal machine learning extends that question:

For whom would it change—and by how much?

Companion Empirical Project

The full Python workflow, including exploratory analysis, propensity-score overlap, ATE estimation, Causal Forest DML, heterogeneity analysis, and T-learner robustness checks, is available in the companion GitHub repository.

Takeaway

Average treatment effects remain useful. But when treatment responses are heterogeneous, the average may hide exactly the variation that matters for policy or product decisions.

Causal machine learning gives us powerful tools for investigating that heterogeneity. The important word, however, remains causal.

Identification comes first. Flexible estimation comes second.