What Judges Don't Say: The Challenge to Content Analysis

First of all, many thanks for the opportunity to guest blog here.  I've found ELSblog to be a valuable resource for information and an invaluable resource for thought-provoking reading.  I'm generally a "lurker," but guest blogging will force me out from the shadows.

Like many others, I was trained as a lawyer and practiced law before I became an academic.  My interest in empirical legal scholarship arises in part from experiences I had and things I observed as a law clerk and as a litigator. 

One of the great frustrations lawyers sometimes experience is the sense that judges are not paying attention to the specific facts and evidence of the case. And in some areas of law, at least, academics also take note.  In the world of employment discrimination law, for example, one need not look hard to find lengthy discussions and critiques of courts' inconsistent, unpredictable, or just plain wrong application of the summary judgments standards to prevent plaintiffs from taking their evidence to a jury or, in some cases, to grant judgment as a matter of law and reverse jury verdicts in plaintiffs' favor.   But these are complaints about the application of law to fact, not about inaccurate reporting of the facts themselves.

At the same time, numerous academics have undertaken important efforts to identify the facts that appear to matter to judges in different contexts. To name but two examples:  Jeffrey Segal's well-known work on search and seizure looks at the relationship of certain facts (whether there search was warrantless, for example, or whether it was a search of a home or a car) to outcome in the Supreme Court's 4th Amendment case law.  Lauren Edelman and others are studying how trial and appellate court judges in employment discrimination cases refer to employers' anti- discrimination policies and programs.  But what these efforts do not account for, by definition, is what the judges do not say, what they do not mention in their summary of the facts.

Of course, some might say, who cares what the judges don't say.  What they do say is what was salient to them, and what they do say is what often becomes important in the case law.  (This latter point is one of the arguments of Edelman et al.)  What they don't say may matter to the disappointed party and lawyers, but it is of no consequence otherwise.

But it seems to me that the question of whether judges -- consciously or unconsciously -- leave out (arguably) relevant facts is directly relevant to several points of great interest to scholars as well as to lawyers.  First, it is directly relevant to the question of how judges decide cases.  Second and relatedly, it may shed light on the extent to which they are political in their decisionmaking.  Third, it should, I think, force hard thinking about the normative question of what we want judges to do.  Do we want them to make predictions about how juries will decide?  Do we want them to announce rules of law applicable in the future?  Do we want them to focus scrupulously but narrowly on the case before them?  Are these goals inconsistent with each other?

The answers to these questions may vary depending on the case, the court, and the issue, of course.   Chief Justice Roberts has recently made some news with his call for more narrow, focused decisions -- and evoked the criticism that the Supreme Court's role requires a broader, less case-specific approach.  In fact, in my view, the question of whether judges accurately report the facts is more important in the trial and appellate courts than in the Supreme Court. 

If I'm overlooking something, of course I'd welcome references to work that does try to address the question of what gets left out of judicial opinions.   But as we in empirical legal scholarship are struggling with the question of how to adequately take into account of what is in judicial opinions (how to operationalize the law), let's not forget that opinions must distill mountains of evidence and piles of briefs into a few pages of summary.  Things must fall through the cracks.  I'm wondering -- is there a pattern to what gets left out and do those things matter?

Comments

[Mark Hall]

In our paper "Systematic Content Analysis of Judicial Opinions," Ron Wright and I catalogue several instances of different researchers who attempt to grapple with this issue, as follows:

"Researchers can specifically examine the fidelity of reported facts, looking for indications of distortion or incompleteness, to determine if the facts are close enough to reality for use in statistical analysis. Hillman, for instance, in his prominent study of promissory estoppel, read a sample of cases more closely looking for any indications that courts stated there was reliance when there really wasn’t. He reports there was little evidence of misrepresentation of the facts, but he does not explain how he made this determination or why he felt he should be able to detect misrepresentation. Krawiec and Zeiler had students code cases both for whether the judge indicated the factor was present, and whether the student thought it was present, in order to “minimize the impact of conscious or unconscious judicial mischaracterizations of fact.” Only “rarely” did they find disagreement between the judge and the student coder. One interesting technique, used in four studies, is to compare facts reported in an appellate opinion with those reported in either the trial court’s opinion or a dissenting opinion. [Richards & Kritzer; Posner; Ignagni; Lawlor.] These researchers found few instances of disagreement or distortion. Lawlor is especially thorough. Using student coders, he tested for 30 possible facts in two Supreme Court cases that each had three opinions. Out of these 60 observations that provided 180 possibilities for discrepancies to appear, he found only two."

For full citations, go to: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=913336.

[CarolynShapiro]

I have indeed thought about comparing trial and appellate court opinions, and I think that might be very fruitful. At least I think it would be fruitful in terms of evaluating how reliable the trial courts' reports of the facts are. I think that the problems and challenges are similar to those I mentioned in my third post about whistleblowing dissents. For my purposes, an appellate court opinion that relies on facts the trial court ignored could be seen as a whistleblower too. I don't think, however, that these comparisons would shed as much light on how reliable the appellate court's reporting is, and that is, frankly, what I'm more interested in.

[Robert Mahnke]

With regard to the question of which facts are included and which facts are omitted, it might be much easier to find something interesting if one compared trial court decisions with appellate decisions in the same cases -- where the appeals courts should be accepting the facts found below -- rather than looking at trial court decisions -- where judges may well ignore the evidence in the course of rendering a decision. I've experienced both as a litigator. You get a much more sympathetic response complaining about the former than the latter, both because it's more egregious -- clearly eroneous vs. a question of discretion -- but also because to fully appreciate the latter, you kinda had to be there.

Carolyn, surely this has already occurred to you.

[Tracy Lightcap]

Ah, ye olde non-decision problem! Admittedly this is a bear to contend with. Might I suggest another way to look at the conundrum?

The main problem here is to see what never reaches the table. One way to get around it might be counterfactuals. We are just looking at a fine example of it - Gaventa's Power and the Powerless - in my senior seminar. The way it might work here is to think of the kinds of issues that might arise on appeal in a particular case with existing precedential constraints. Then one can look at two things. First, what issues that arise on appeal are ignored and, second, what issues never arise, despite the expectation that they would, given the facts in play? This tracks Lukes's second and third dimensions of power. The issues that are ignored are obviously ones the justices don't want to make decisions about, for one reason or another. Those that never arise, although one might expect them to, are issues that have been socialized out of consideration.

How to approach these empirically? History is the answer. The first kind of non-decision is usually the product of a set of path dependent cumulative positive feedback loops that lead the judges to feel they can ignore the issue and the attorneys to feel that they at least tried, despite not having much expectation that they would prevail. The second is usually deeper and involves patterns in the professional socialization of the parties and long standing cultural barriers to raising issues in law. Or, to be more exact, so I would suppose.

Well! There's a nice, long, involved research project for someone younger and better funded. I'm always coming up with those.

[Sara Benesh]

I don't have much evidence to bring here, but I have read a lot of confession cases in the circuit courts, attempting to ascertain whether those judges mention the "right" facts in deciding their cases ("right" meaning the ones the Supreme Court says they should consider), so I thought I'd weigh in. I coded the cases trichotomously to attempt to ascertain whether they are paying heed to the High Court by considering all relevant facts; so, each fact is coded as "mentioned as present," "mentioned as not present," and "not mentioned." In so doing, it seems to me that the lower courts DO mention every conceivable fact that may be relevant (to the Supreme Court) in determining whether the confession is voluntary or not, so they don't appear to be ignoring facts (at least not in confession cases in the circuit courts). This still leaves us with a problem, though -- how do different judges "weight" different facts? It's surely possible (probable?) that they are interpreting the facts while discussing them, emphasizing those they deem to be dispositive, de-emphasizing those anithetical to their preferred disposition.

It seems like these are related problems and I agree that they are important to attempt to understand. Too bad I have no answers. But thanks for the post, Carolyn! I'll look forward to seeing what you come up with.

[Josh Fischman]

I agree that this is a great question. We really ought to know if/when judicial opinions are a reliable source for the facts of a case. If they aren't, that would have some profound implications for empirical research.

In courts where cases are randomly assigned to judges, the presence of case characteristics should be uncorrelated with the ideology of the judge. So you could easily compare different kinds of case characteristics to see if this is true. You might also want to think carefully about which kinds of case characteristics would be more or less susceptible to manipulation.

[frank cross]

I think you're on to something important. This is potentially amenable to study by examining the briefs (or lower court opinions). That takes a lot of work, though.

An easier study that I'm playing around with examines justice's choices of what precedents to cite, using the briefs as a baseline