Puzzle 2: Propensity Score Matching

Question 1: How popular is Propensity Score Matching in the ELS literature?

Endogeneity and sample selection bias are major concerns in any empirical study of legal institutions. We rarely work with experimental data, in which subjects (people, firms, states, courts) are randomly assigned to "treatment" and "control" groups. Our data are instead taken from environments in which the subjects choose their own groups: young doctors choose whether to locate in states with ("treatment") or without ("control") medical malpractice caps; litigants choose whether to pursue claims to trial ("treatment") or settle out of court ("control"). Our data are also taken from environments in which the outcomes of interest (e.g., crime rates) may be driving the legal institutions (stringent laws or large police forces), not the other way around. These dynamics generate self-selection bias, sample selection bias, simultaneity bias (or “reverse causation”), etc.

There are, of course, many tools for addressing these biases. Common techniques include before-after estimators such as difference-in-differences (DD) and cross-section estimators such as instrumental variables (IV) and the Heckman selection estimator (Heckit). In recent years, DD (and triple DD) has become common in L&E papers and elsewhere.

In economics and other literatures, another technique is gradually gaining popularity: propensity score matching (PSM). This technique offers a method for structuring non-experimental data to look like experimental data. For every subject in the "treated" group, the researcher finds a comparable subject in the "control" group. Comparing the two groups, we can get an estimate of the effects of legal institutions on outcomes of interest.

Is this technique becoming popular in ELS circles?

Question 2: Do scholars work as hard to justify PSM as they do to justify IV or other techniques for handling potential endogeneity bias?

PSM is a good technique for dealing with endogeneity problems if the researcher has a very rich dataset and can observe nearly all of the factors driving the potential bias. The researcher, for example, must be confident that she can observe all of the important variables affecting a subject's choice between treatment and control groups. In other words, PSM facilitates causal inference only if selection on unobservables is not an important problem.

The trouble, it seems to me, is that ELS scholars can rarely be confident that their datasets are rich enough to rule out selection on unobservables. I’ve rarely attended a workshop where the audience members were convinced that unobservables were unimportant. This is why every empirical scholar is on a holy-grail quest for valid instruments or a convincing DD framework that minimizes the "selection on unobservables" problem. To be sure, these techniques require their own leaps of faith and must be justified carefully. IV estimation, for example, is compelling only if the researcher can persuade readers that the chosen instruments are (i) correlated with the subjects' choice between the treatment and control groups but (ii) uncorrelated with the outcome of interest. This is a tall order, which explains why scholars like Steven Levitt are famous for identifying instruments that are largely persuasive.

Comments

[William Henderson]

Ed, this is a terrific post. Can you provide a cite to a study that effectively uses PSM?