I'm a big fan of regression discontinuity studies. In a regression discontinuity study, there is typically a discontinuous determination of who receives a treatment. For example, a cholesterol medication may be given to all individuals with bad cholesterol levels over x, but to no individuals with cholesterol levels below x.
In a regression discontinuity design, we exploit the discontinuity to determine the treatment effect. The idea is that some individuals randomly fall just above the treatment cutoff (in my example, they have a cholesterol level of x+epsilon) while others fall just below (have a cholesterol level of x-e). These individuals are pretty similar-- their cholesterol levels are almost identical, yet they get treated differently. Some get the treatment, but others don't. If, after receiving the treatment, a group of people with original cholesterol levels just above x have a much lower cholesterol level than a group with original cholesterol levels just below x (who do not get the medication), then we can attribute that decrease in cholesterol level to the medication.
It is, of course, true that the people who receive the medication have a higher average cholesterol level than those who do not, but the size of the effect is small and can be controlled for.
When regression discontinuity assumptions are valid, then the design is a great way to obtain valid inferences of causal effects in a non-experimental setting. For a really cool application of regression discontinuity to estimate the impact of class size on test scores, see Angrist and Lavy (1999).
The validity of the identification from regression discontinuity designs comes from the assumption (among others) that people with cholesterol levels just below x are very similar to those with a cholesterol level just above x. This assumption will be satisfied when people are unaware of the cutoff, or if the cutoff is not important enough to induce any change in behavior. When making or not making the cutoff is an important choice variable, however, then the validity of the inferences may be flawed.
For example, suppose that people know x and that this medication is highly anticipated-- some people really want to be able to take the medication. Suppose also that you can regularly check your cholesterol and affect your level by modifying behavior. If this is the case, then people with cholesterol levels just below x might be very different from people with cholesterol levels just above x. The ones with cholesterol levels just above x might be people who really wanted the medication and made sure to have just enough cholesterol, but not more than that. The ones with cholesterol levels just below x apparently didn't care about getting the medication or were "bad" at controlling their cholesterol levels. Under these assumptions, the groups are very different in their attitudes towards medication and/or their motivation levels, in spite of the fact that their cholesterol levels are very similar.
When we apply the regression discontinuity design in this case, then we do not get the valid inferences of the impact of the medication that we'd like. The difference in post-treatment cholesterol levels between those with original cholesterol levels just above vs. just below x may be due to the medication, but they also may be due to other factors, such as differences in motivation or difference in belief in the power of medication.
At the recent AFA meetings in Chicago, there was a very interesting paper by Chava and Roberts that may be subject to this critique. The paper tries to estimate the impact of violation of debt covenants on the amount of corporate investment. The paper applies a regression discontinuity design. Many covenants define minimum net worths. The paper estimates the effect of violating the covenant by comparing the investment decisions of companies that barely violated net worth covenants with the decisions of companies that barely avoided violations. The paper assumes that these companies are similar except for the fact that some violated the covenants and others didn't.
I'm not so confident that this assumption is valid. I presume that most companies want to avoid covenant violations. Once you've avoided a violation, however, then you care much less about your actual net worth and don't spend much time worrying about it. This means that companies that barely violate the covenants may be systematically different than those that don't violate the covenants. The companies that violate may less careful, more honest, or be in a much more delicate financial position than those that are just above the minimum new worth. The effect found by Chava and Roberts may be the impact of covenant violations, or it may be due to the fact that the two groups they are comparing are systematically different.
I don't want to sound too negative, however. Regression discontinuity is a great research design, and I'd love to see more of it in empirical legal studies.