After Mark and I collected 125 examples of content analysis, we evaluated how well scholars used this technique over the years. As you might expect, authors of the studies have paid attention to the selection of judicial opinions for study. But we think authors are worried about the wrong selection problem: it is common to see discussions of selection bias, instead of more troublesome issues of selection replicability.
Many scholars over the years have discussed selection bias. They wonder if the published opinions they reviewed were typical of the entire pool of written judicial opinions; whether the chosen opinions were typical of judicial decisions more generally; whether the chosen opinions were typical of litigated cases; whether the chosen opinions were typical of underlying legal problems in the world. The connection gets weaker at every step, but you saw that one coming, right?
The proper inferences to draw from a collection of cases must account for these gaps. Still, the selection bias problem is actually smaller for judicial opinions than for many other texts to be studied through content analysis -- the number of relevant opinions is often small enough that the author can code the entire relevant universe.
A selection issue that get much less attention, however, is selection replicability. Most often, these content analysis projects describe the general search criteria that the authors used to identify cases, and just stop there. A smaller group specify the sources searched (e.g., West Decennial Digests for older projects, designated LEXIS/Westlaw databases for newer projects). A small number of projects list the actual Westlaw/LEXIS search terms used. A surprising number of the projects simply listed the cases that were included in the study.
All too often, the description of the case selection method was too sketchy to allow another researcher to replicate the same results. (A simple listing of the cases doesn't exactly do the job.) That basic building block of the scientific method -- replicability -- deserves more attention than the now well-known potential biases that flow from studying the world through a collection of judicial opinions.
It's a important shift in thinking that still hasn't percolated down very far into the legal academic ground.