Empirical Legal Studies

Multidimensional Scaling - Blumenthal #5

Another useful methodology often used in psychology -- less in social, more in cognitive -- is one alluded to here before, "multidimensional scaling" (MDS).  MDS has only been used in a small number of legal analyses, but can be quite helpful in the right context.

As I describe in a forthcoming paper, "MDS is a procedure that helps researchers uncover 'hidden structures' in data by graphically plotting respondents' perceptions of perceived similarities (or dissimilarities) among various stimuli.  When these stimuli are located on a plot based on such perceptions, underlying dimensions that respondents may have used (consciously or not) can be inferred."

So, for instance, a bunch of stimuli are identified -- crimes, Supreme Court opinions -- and every pairwise combination is created.  Each pair is rated on how similar or close they are (though the researcher usually does not articulate the scale to be used, just asks "how similar"?).  Those similarity ratings are then plotted to give the best fit to the data, yielding a "map" of the ratings that also reflects the perceived structural relationship of all those stimuli.  Finally, by examining the map, one might be able to identify the dimensions that, implicitly or explicitly, underlay subjects' ratings (and also, interestingly, look at individual differences, e.g., do men and women or Republicans and Democrats come up with different maps or weight those dimensions differently?).

As a specific, somewhat prosaic example: subjects might be asked to rate the similarity of countries in the world, and a plot made of the countries based on those similarity ratings.  The resulting map might look like a map of the world, which could suggest that subjects were, implicitly or not, using dimensions involving distance, in particular, dimensions of north/south and east/west.  But the plot might line up the countries on a single axis going from largest to smallest, or richest to poorest, or most to least economically developed, suggesting that subjects used that single dimension in their ratings.  Inferring which dimensions are important to respondents helps understand the way they see the world (literally, here), and what factors are important to them.

In the legal context, I used it in that paper to map people's perceptions of the seriousness of crimes.  MDS is broadly useful for looking at lay intuitions about legal institutions, punishments, or cases; or to assess whether different punishments may have been proportionate; or any context in which (a) similarities among stimuli may be interesting or (b) we are interested in understanding what the basis might be for people's judgments or perceptions.

A classic book on MDS is by Kruskal and Wish, from the 1970's.

Insights from the Methodology of Psychology (Blumenthal #4)

Picking up on the methodological contribution I think psychology can make to ELS, it seems as though psychology emphasizes experimental work to a greater degree than current ELS. Discussing the excellent methodology workshop that Lee Epstein and Andrew Martin run, someone recently joked that it covers 95% of all the methodology ELS scholars need to know. I’d quibble with that if, as I understand (without having attended), there’s little coverage of experimental design and analysis (e.g., analyses of variance or ANOVA’s). To be very clear, all I mean by that is I think the experimental approach could be a larger proportion of ELS work than it currently is.

Experimental research, of course, can contribute greatly to our understanding of a number of topics and phenomena (jury understanding of sentencing instructions; eyewitness accuracy; racial discrimination by judges and jurors; effects of policy interventions; different influence of legal procedures both in and out of the courtroom, etc., etc.). Researchers can manipulate specific variables and test their causal effect, tweaking particular aspects to really parse out specific effects that may be more precise than multiple regression. In some instances – when a researcher is working with existing databases – experiments can be less constrained by data limitations, can be more proactive in data collection.

Among the downsides, of course, is the perception that experiments aren’t “externally valid” – they don’t sufficiently reflect the real world that we can profitably generalize. Sometimes, true. But they sometimes do, and sometimes can be conducted in real-world settings. Other times, finding an effect in the lab suggests it might be even stronger in the real world: the example often given is with comprehension of capital sentencing instructions – when highly educated mock juror subjects from “elite universities” don’t understand them, how much more of a concern might it be for less educated jurors faced with such instructions in the real world? In yet other contexts, again with mock juries, differences between undergraduate samples and community member samples are substantially less than critics have suggested (Bornstein, 23 Law & Hum. Behav. 75 (1999)).

I don't want to oversell the experimental approach, but it could certainly be another arrow in the ELS quiver.  So to speak.

Blumenthal #3 - More on Psychology

As a follow-up, I would point out some excellent psychology-and-law resources (sorry; now that I teach it at a law school, it's "law-and-psychology" - sort of emphasizes the point, doesn't it).

First is the American Psychology/Law Society, founded in the late 1960's; with annual conferences highlighting the most current and topical research in legal psychology.  The website has links to various newsletters of the Society, which include conference programs in order to see who is doing what sort of empirical research.

Second is the Society-sponsored, peer-reviewed journal, Law and Human Behavior, published by Springer.  Publishing top-quality empirical research, LHB is consistently one of the most highly-cited non-law-review journals.  The topic areas are often limited to the "traditional" psych-and-law topics I've mentioned before, though - forensics, eyewitness, juries - and I am sure that its editors would welcome a broad range of submission topics of the sort that ELS people here investigate.

Both APLS and LHB are good resources for finding primary research and empirical, interdisciplinary collaborators (JD/PhD's are not uncommon in APLS) - IMHO, both can help as the ELS field continues to develop.

Blumenthal # 2 - A Plea for Psychology (with apologies to John Austin)

A Plea for Psychology (with apologies to J.L. Austin)

So I've gotten off to a slow start, but I'll try to make some contribution now.

As I mentioned, I wonder if psychology as a discipline has gotten short shrift from ELS. As a historical matter I think so; research in legal psychology has been conducted, off and on, for about 100 years. But until recently the legal system has been skeptical, suspicious, at times hostile to psychology, and there’s not much discussion of psychological research in modern ELS – with exceptions, much of the work stems from economics or political science.

There are historical reasons for this skepticism. The conventional story is that psychology and law as a discipline was founded around 1908, with the publication of Hugo Munsterberg’s On the Witness Stand, documenting his efforts to persuade the legal system that his empirical work in criminal justice – eyewitness research, memory, perception, lie-detection, etc. – was relevant and important. (Another interesting contribution, written in a more scholarly vein, was a book by G.F. Arnold, a British legal scholar, Psychology Applied to Legal Evidence (1906)).

Munsterberg's book was pugnacious and had a lot of 'tude -- probably too much ("The time for Applied Psychology is surely near. . . .  The lawyer alone is obdurate;" "The lawyer and the judge and the juryman are sure that they do not need the experimental psychologist. . . .  They go on thinking that their legal instinct and their common sense supplies them with all that is needed and somewhat more.")  Hugo's book prompted a now-classic response article by John Henry Wigmore, in which Wigmore staged a mock lawsuit accusing Munsterberg of libeling the legal profession.  Both tongue-in-cheek and vitriolic, Wigmore's cross-examination devastated poor Hugo, and as a result, one commentator has said, for years American psychologists "left the law rather severely alone."  Last week a wonderful conference was staged at John Jay College of Criminal Justice, though, celebrating OTWS's centenary: Off the Witness Stand: Using Psychology in the Practice of Justice."  With a keynote speech by Janet Reno, and talks by many top researchers in in psychology discussing the state of psycholegal research today--experts in eyewitness testimony, memory, jury decision-making, forensic psychology, coerced confessions, and more--it was geared toward both scholars and practitioners, and was a good state-of-the-discipline weekend.  (One concern, as I hope to discuss in another post, is the impression that criminal justice is all there is for psychology to look at.)

And there are methodological / institutional reasons for the skepticism.  As I and many others have noted, experimental psychology takes a different approach to finding answers: objective rather than adversarial, nomothetic rather than idiographic, probabilistic rather than certain, descriptive rather than prescriptive.

But this skepticism is unfortunate, and has led to a lack of communication between the disciplines of law and psychology.  I think ELS suffers from that lack, in at least two ways.  The first is simply mutual unfamiliarity with the other field’s research.  This is unfortunate historically (for instance, the explosion of heuristics-and-biases research over the last 10 years or so, which builds on psychologists’ work that has been around for 30 years and appears in any introductory cognitive psychology textbook), but also in terms of keeping up with empirical developments in psychology (and other social sciences too, of course).

*

The second, which I'll turn to next post, is methodological. . . .

*

For those who are still here, thanks!

Blumenthal Post #1

I appreciate the invitation to participate in the ELS blog.  As Jason mentioned, my empirical background is in psychology, so I’ll try to highlight a number of insights from work in that field that inform empirical – and experimental – legal studies, as well as point to some interesting ongoing work in psychology and law.  I’ll also mention a few methodological issues I think get neglected sometimes in our ELS discussion, hoping to expand the empirical approach we take to such study. And if I haven’t yet bored everyone by then, I’ll try to come up with other ELS bits as well.  Thanks again!

Guest Blogger: Jeremy A. Blumenthal

This week's guest blogger is Jeremy Blumenthal, Assistant Professor of Law at Syracuse University College of Law.   He earned his A.B., A.M., and Ph. D. in Social Psychology from Harvard University. He earned his Juris Doctor and graduated cum laude from the University of Pennsylvania Law School, where he was Editor-in-Chief of the University of Pennsylvania Journal of Constitutional Law and received several writing prizes. Most recently, he was a Faculty Fellow at Seton Hall Law School, teaching 1L Property and Law and the Social Sciences.  Previously, he clerked for the Honorable Maryanne Trump Barry of the U.S. Court of Appeals for the Third Circuit, and was an Associate for Latham & Watkins, LLP, in the Newark, NJ office.

Professor Blumenthal has published widely on topics in law and psychology, including demeanor evidence, the reasonable woman standard, victim impact statements, and more generally on the role of social science in assisting the legal system. He has also published in the areas of evidence and comparative law. His most recent publications focus on the role of emotions in the legal system; current research projects address topics in capital punishment, empirical work in law and psychology, and topics in property theory.

Numbers in the Classroom

It's been fun guest blogging this week, but this is likely my last post.  I'm going to wade into the ongoing discussion here and elsewhere on what kinds of statistical and social science background law students should get (or be offered) and lawyers need.  The easy answer to those questions, not likely to be controversial here, is more than they get!  Perhaps most useful to the students are those classes that focus on the types of numeracy and related skills that lawyers need when dealing with expert witnesses. 

But I don't think that teaching about such matters should be restricted to such classes or to classes covering, for example, empirical studies of judicial decisionmaking.  In my employment law class, for example, I cover the law governing the use of polygraph tests and other kinds of screening tests by employers.  While I'm at it, however, I walk the students through a (very simple) set of exercises aimed at helping them understand the statistics of screening tests.  What does it mean to say that a test is 90% accurate, or that it has a specificity of 95% but a sensitivity of only 70%?  Why might you want to know something about the prevalence or expected prevalence of whatever it is you are screening for?  How do you interpret a "positive" result?  Invariably, most of my students find this discussion extremely intimidating and even panic-inducing.  ("I didn't realize we'd have to do math in this class!" or "Is this going to be on the final?" are the two most common responses.)  Yet an employment lawyer advising a client on whether or not it is worth the legal hassles or potential hassles associated with using and relying on different types of screening tests simply cannot give competent advice without knowing how to think about these questions -- or at least without knowing that those questions exist and need to be answered.

I'm sure most readers of this blog wouldn't disagree and are likely to find ways to include numeracy if they teach doctrinal classes. I'm also  sure many of our colleagues who do not consider themselves ELS devotees do so as well.  It would be interesting to know how much this happens or, to the extent it doesn't, do law professors feel that more would be helpful but may not themselves feel competent in presenting the information.  I believe that statistical and social science information closely related to the doctrinal subjects already taught may be the easiest to "sell" to faculty and students alike -- certainly easier sells than, say, empirical studies of judicial decisionmaking or general statistics classes that are not directly tied to application in they study of doctrine or the practice of law.

Just the Facts?

Thanks to everyone who has commented on my posts so far.  The responses to my first post, on whether judges accurately report the facts, were extremely helpful.  I think that Sara Benesh may have put her finger on one of the real challenges of doing this kind of work – deciding what facts should have been reported. In situations where the Supreme Court has made it clear that certain types of facts must be taken into account, whether by incorporating those facts explicitly into a legal test or standard or by doing so implicitly by relying on such facts (or their absence) repeatedly, my hypothesis would be that the trial and appellate courts do a pretty good job of accurately reporting them – as Sara seems to have found. 

But my interest in this thorny question arises from a different context.  In employment discrimination cases, courts often must decide whether to grant summary judgment.  So they have to evaluate whether a reasonable jury could find for the plaintiff.  Every case presents a different constellation of facts and much of what courts must do is to weigh the facts.  Although the Supreme Court has provided a framework for this analysis (McDonnell-Douglas), in fact what courts do here is quite case-specific (and, as I’ve already suggested, quite unpredictable).  Judges in these cases don’t report every fact or piece of evidence.  They can’t.  And presumably, the decision to omit a fact is based on a determination that the fact, or piece of evidence, is not material.  So if a researcher were to go through the briefs and identify the facts mentioned by the lawyers but not the court, the researcher would still have to make a judgment whether the facts should have been mentioned.

So how to get around this problem?  One possibility is to focus on dissents.  In some cases, the dissents blow the whistle.  In Malacara v. City of Madison, for example, Judge Williams in dissent lays out the plaintiff’s substantial evidence to support his claim of discrimination, evidence that the majority simply ignores.  The problem is that such whistleblowing will point to only a fraction of the cases in which this problem arises, either because there is no dissent or because a draft dissent prompted discussion of the previously omitted facts.  This latter situation may be interesting for other reasons – does the presence of certain judges and the increased threat of whistleblowing affect the way or which facts are described?  This is a panel effect that I do not believe has been studied, but would be very difficult to observe, at least in the absence of judges’ papers.

There’s an additional facet to this problem that has to do with how the law develops, and that might provide some insight.  My colleague Mark Rosen has coined that expression “rulifying” to describe the way that the Supreme Court reliance on facts can turn those facts into elements of a legal standard or rule.  Putting these ideas together suggests an interesting possible empirical approach: is there less “whistleblowing” in areas of law that have relatively well-developed sets of facts courts are required to address versus areas of law that require judges simply to rely on their overall assessment of the evidence or facts – the totality of the circumstances?

What Judges Don't Say: The Challenge to Content Analysis

First of all, many thanks for the opportunity to guest blog here.  I've found ELSblog to be a valuable resource for information and an invaluable resource for thought-provoking reading.  I'm generally a "lurker," but guest blogging will force me out from the shadows.

Like many others, I was trained as a lawyer and practiced law before I became an academic.  My interest in empirical legal scholarship arises in part from experiences I had and things I observed as a law clerk and as a litigator. 

One of the great frustrations lawyers sometimes experience is the sense that judges are not paying attention to the specific facts and evidence of the case. And in some areas of law, at least, academics also take note.  In the world of employment discrimination law, for example, one need not look hard to find lengthy discussions and critiques of courts' inconsistent, unpredictable, or just plain wrong application of the summary judgments standards to prevent plaintiffs from taking their evidence to a jury or, in some cases, to grant judgment as a matter of law and reverse jury verdicts in plaintiffs' favor.   But these are complaints about the application of law to fact, not about inaccurate reporting of the facts themselves.

At the same time, numerous academics have undertaken important efforts to identify the facts that appear to matter to judges in different contexts. To name but two examples:  Jeffrey Segal's well-known work on search and seizure looks at the relationship of certain facts (whether there search was warrantless, for example, or whether it was a search of a home or a car) to outcome in the Supreme Court's 4th Amendment case law.  Lauren Edelman and others are studying how trial and appellate court judges in employment discrimination cases refer to employers' anti- discrimination policies and programs.  But what these efforts do not account for, by definition, is what the judges do not say, what they do not mention in their summary of the facts.

Of course, some might say, who cares what the judges don't say.  What they do say is what was salient to them, and what they do say is what often becomes important in the case law.  (This latter point is one of the arguments of Edelman et al.)  What they don't say may matter to the disappointed party and lawyers, but it is of no consequence otherwise.

But it seems to me that the question of whether judges -- consciously or unconsciously -- leave out (arguably) relevant facts is directly relevant to several points of great interest to scholars as well as to lawyers.  First, it is directly relevant to the question of how judges decide cases.  Second and relatedly, it may shed light on the extent to which they are political in their decisionmaking.  Third, it should, I think, force hard thinking about the normative question of what we want judges to do.  Do we want them to make predictions about how juries will decide?  Do we want them to announce rules of law applicable in the future?  Do we want them to focus scrupulously but narrowly on the case before them?  Are these goals inconsistent with each other?

The answers to these questions may vary depending on the case, the court, and the issue, of course.   Chief Justice Roberts has recently made some news with his call for more narrow, focused decisions -- and evoked the criticism that the Supreme Court's role requires a broader, less case-specific approach.  In fact, in my view, the question of whether judges accurately report the facts is more important in the trial and appellate courts than in the Supreme Court. 

If I'm overlooking something, of course I'd welcome references to work that does try to address the question of what gets left out of judicial opinions.   But as we in empirical legal scholarship are struggling with the question of how to adequately take into account of what is in judicial opinions (how to operationalize the law), let's not forget that opinions must distill mountains of evidence and piles of briefs into a few pages of summary.  Things must fall through the cracks.  I'm wondering -- is there a pattern to what gets left out and do those things matter?

Guest Blogger: Carolyn Shapiro

Our guest blogger this week is Carolyn Shapiro, Assistant Professor of Law at the Chicago-Kent College of Law.  Professor Shapiro earned a B.A. with general and special honors in English from the University of Chicago, an M.A. from the University of Chicago Harris Graduate School of Public Policy and a J.D. (high honors) from the University of Chicago Law School, where she was articles editor of the University of Chicago Law Review and a member of the Order of the Coif.             

After graduation, Professor Shapiro was a law clerk for Chief Judge Richard A. Posner of the U.S. Court of Appeals for the Seventh Circuit and  for Justice Stephen G. Breyer of the U.S. Supreme Court.             

Professor Shapiro's scholarly interests include federal courts and labor and employment law.  She teaches professional responsibility, employment law, and legislative process.