Random Variability and Inference

Slide 1 of 8

Two Types of Uncertainty in Causal Inference

Every estimate of a causal effect is uncertain. However, not all uncertainty is created equal. In causal inference, we must distinguish between two fundamentally different types of error: random error (due to sampling variability) and systematic error (due to biases in study design or analysis). Understanding this distinction is crucial because the two types of error require completely different solutions.

Random Error (Sampling Variability) Random error arises because we observe only a sample, not the entire population. Even if our study design is perfect, the estimate we obtain from one sample will differ from another sample due to chance.

Decreases as sample size increases (proportional to $1/\sqrt{n}$)
Can be quantified using standard errors and confidence intervals
Is "symmetric"—equally likely to make estimate too high or too low
Is the focus of traditional statistical inference

Systematic Error (Bias) Systematic error arises from problems in study design or analysis that cause our estimate to consistently deviate from the truth. Even with infinite sample size, the estimate would still be wrong.

Does NOT decrease with larger sample size
Cannot be quantified from the data alone—requires external information
Is "directional"—tends to push the estimate in one direction
Includes confounding, selection bias, and measurement error

Characteristic	Random Error	Systematic Error
Cause	Sampling variation	Study design flaws
Solution	Larger sample size	Better design, adjustment, sensitivity analysis
Quantification	Standard methods (SE, CI)	Requires assumptions or external data
Direction	Unpredictable	Often predictable
With N → ∞	Disappears	Remains

The Hierarchy of Threats In most observational studies, systematic error (bias) is a greater threat to validity than random error. A large study with confounding will give you a very precise wrong answer. Yet traditional statistical training focuses almost exclusively on random error!

Reference: What If, Chapter 10 "The confidence interval reflects only random variability. It does not incorporate systematic sources of error such as confounding, selection bias, or measurement error" (Hernán & Robins, p. 123).

This chapter marks a transition: after learning about specific biases (confounding, selection, measurement error), we now step back to understand how they fit into the broader framework of uncertainty. The key insight is that standard statistical inference addresses only half the problem—and often the less important half.

Use ← → arrow keys to navigate