Getting Data: My Experience with the California Bar

Today I am in California for a hearing before the Board of Governors, which is the elected body that oversees the state bar exam.   For the past 18 months, I have been part of a research team (with Richard Sander, Vik Amar, Doug Williams, and Stephen Klein) seeking California data to test a possible mismatch effect in law schools.  The substance of the hearing today (see agenda here) was to determine whether we would be permitted to obtain regression results, via the Bar's own longtime psychometrician, from the Bar's rich archive of historical data.

After hearing approximately 50 minutes of  prepared testimony from those for and against our study, the Board voted to deny our data request.  Based on the tenor of the meeting, which was set by a well-organized opposition led by Cheryl Harris (UCLA Law) and Michele Landis Dauber (Stanford Law), the vote turned on legal uncertainties surrounding candidate confidentiality and consent. 

Acceding to these concerns has, in my opinion, three major flaws: (a) 0ur research team would only have access to regression tables with results aggregated among groups of schools, so no individually identifiable information would ever be released to researchers or the public;  (b) we would, at all times, be subject to university IRB protocols; and (c) the broad construction given to consent in this context suggests that much of the research sponsored by the California Bar over the last 20 years has been unlawful; moreover, this decision casts doubt on the scope of any future research by the California Bar. 

[Note: Richard Lempert (Michigan Law and formerly chair of the NSF Law & Social Science Program), wrote this memo on the IRB issue, which he graciously forwarded to our research team in advance of the hearing. Rick's assessment, however, is quite different from my own experience in obtaining IRB approval at three different institutions for projects which raised, in my opinion, much more significant human subjects issues.  Regardless, no research would occur without IRB approval; that should be obvious to all parties involved.]

Of course, the Board of Governors is not an adjudicative body -- rather, in this case, it is the regulatory unit that would be dragged into court if it approved our proposal.  Thus, on one level, I can understand the institutional reasons for taking a cautious approach.  (For a snapshot of the public debate leading up to this decision, see this National Law Journal op-ed by Cheryl Harris and Walter Allen, and our NLJ response; in the spirit of transparency, all documents related to our projects are posted on this website.)

Although I am disappointed with the outcome, I am proud of the testimony we put on today.  The basic theory underlying the mismatch effect is that recipients of large preferences (regardless of race) tend to learn less because the classroom instruction is pitched toward a modal student with significantly higher entering credentials.  Honest researchers can argue over the methodology and inferences of Sander's Systemic Analysis study.  But existing data provide little comfort that our current system is working well.  Minority students are disproportionately clustered at the bottom of their class.  Since LGPA is the single best predictor of bar passage, it is not surprising that these same students struggle on the bar examination. Sander's first choice/second choice analysis (in his 2005 Stan. L. Rev. response essay) suggests that law school performance and bar passage prospects actually increase when a student opts for a less elite law school. In my opinion, if this theory is wrong, it is imperative that it be debunked empirically rather than through efforts to withhold data.

As a lifelong Democrat and ardent supporter of racial diversity, I don't need to apologize for raising these issues.  As a said in my prepared remarks, it does not follow that evidence of a mismatch effect requires the dismantling of racial preferences -- indeed, in my opinion, this would be a very bad idea. 

But if we fail to diagnosis factors that contribute to low minority bar passage, we have no basis to formulate effective policy or educational strategies.  Regardless of which way the data cut, our study would have guaranteed one of two favorable outcomes:

1. Using the more refined California Bar exam (i.e., a continuous dependent variable rather than pass/fail and schools put in analytically useful clusters, unlike the LSAC-BPS data), the mismatch theory would have little or no empirical support.  So a contentious academic theory would be put to bed, at least in the law school context.
2. A mismatch effect would be supported, which would pressure law schools to take concrete steps to help current or prospective students.  These might include: (a) disclosures that reveal bar passage prospects for past students with similar entering credentials; (b) creation of rigorous academic support programs (such as this one) that increase the bar passage for students in the bottom 1/2 of the class; and (c) identify curricular and teaching strategies that produce higher bar exam scores.

As legal educators, we should want more than the status quo.

Comments

[Supremacy Claus]

Is it possible the disparity in pass rates stems from 1) the scientific invalidity of the exam; 2) racial animus by graders; 3) a violation of Larry P. v Riles, no matter the validity or fairness of the grading?

Inside data is useful for litigation. As you said you sought records of aggregate data from a Board that would be named in litigation. The confidentiality argument seems pretextual.

Bill, the Bar uses a .gov web site. It is certainly a quasi-governmental organization in this licensing function. Why did you not assert this law?

http://www.thefirstamendment.org/publicrecordsact.pdf

[Corey]

"Indignation is not part of the equation because it does not move the ball forward."

That's fine but there is more than one game. I share your interest but not your faith in empirical data, and reasonableness has never been my strong suit. Passion and Rhetoric has its place too, and the other side is willing to use it. Sometimes people's assumptions need to be pushed and pushed hard. Objectivity doesn't need my help to rule the day.

I am sorry that this research was stymied by this decision. However, so far Sander's research has been cited to me about 20 times in arguments with conservatives seeking to dismantle affirmative action and thus de facto resegregate the law schools. Sander's research is being used for ill. The Federalist Society wants to invite him to campus to explain why they think they didn't get into Harvard. We have to travel into the world of irrational prejudices and subjective beliefs to see why, but on the street, "mismatch" is sometimes treated as if it is inherant.

Sorry for "scoring points," but there is a legitimate critique that objective data can have subjective harms. Many believe that Sander and Co. have a responsibility to explain their results, so that actual bigots don't come up with "racial differences" as an explanation. I've witnessed people going there. Now I suggest test bias and discrimination as a working hypothesis and a situs for reform. And being somewhat of a revolutionary at heart, I am willing to start the reform before all the data is in. This does not mean that I am not serious or that my approach to this question was not deeply considered. I have an entire ream of printouts of Sander's and his critics papers (of which I like the Coleman/Gulati piece the best.)

[Stuart Buck]

Mr. Lempert --

I've read your memo, but my impression is that Bill Henderson and his colleagues have shown that the purported privacy concerns are makeweight. As for the bulk of the memo -- which addresses concerns that the data could be misinterpreted in a wide variety of ways -- I'm not trying to show that you're wrong on the merits. Nor need I do so. Such concerns can be properly raised later, but do not justify the suppression of data as an initial matter. If data were to be misinterpreted, the answer is to bring forth the proper interpretation -- not to suppress it altogether.

As for this:
"He apparently thinks because I have authored or co-authored several pieces pointing to serious methodological problems in some of Prof. Sander's other work I should be barred from suggesting that there are serious problems with additional research Sander has proposed."

I'm not suggesting a lack of good faith or sincerity, and indeed, you might be right about how the data could be misinterpreted. Still, try to put yourself in someone else's shoes and think about how it looks when a participant in a heated academic debate argues that the other side shouldn't be able to get access to data. Wouldn't you be skeptical if it were your opponents trying to block your research?

[Rick Lempert]

Stuart Block in a comment above suggests that there is something "a bit alarming" about my weighing in on the data access issue in favor of supporting the California Bar Examiner's decision to deny Sander's team access to the bar data it seeks. He apparently thinks because I have authored or co-authored several pieces pointing to serious methodological problems in some of Prof. Sander's other work I should be barred from suggesting that there are serious problems with additional research Sander has proposed. Bill Henderson has thoughtfully provided above a link to my full memo which I hope interested people will read.

For those who don't wish to read as much, my motivation for what I wrote is spelled out in the portion of my statement reproduced below. I wonder which of my propositions Mr. Block disagrees with. He may, of course believe that I did not demonstrate the proposed research was
flawed along the lines I mention, but he did not say that, and even if he felt that way it would not make it true. Readers interested in seeing if my arguments against funding the proposal as submitted are sound and whether I am acting reasonably when I suggest that a substantially revised proposal is needed to show that legal, ethical and methodological hurdles can be overcome, should read the documentation Sander has put on his website, read what I wrote after considering these materials and decide for themselves.
Rick
"Empirical research need not be perfectly done or answer all questions to make valuable incremental contributions. But unsound research, and research that does not meet high standards of scientific integrity, can harm by misleading both the public and policy makers about what available evidence shows. Moreover, even sound and valuable empirical research can do harm if it adversely affects human subjects."

[Bill Henderson]

Corey,

Come on, I know you are a brilliant thinker. But you are just scoring points here. Desegregating Arkansas schools was made possible by a legal claim under the 14th amendment. And going to the moon was political attractive, especially after Sputnik. The only obstacles were the limits and pace of science. Neither one of those situations apply here.

My strategy is simple: get data, do careful research, be reasonable, build political bridges whenever and wherever possible, and let the passage of time and events create an opening for progressive change. Indignation is not part of the equation because it does not move the ball forward.

[Corey]

Oh, and while I agree that trashing the LSAT would be radical relative to what has been possible in today's political climate, it is not such a big deal when judged alongside such prior goals as say... desegregating Arkansas schools or going to the moon.

[Corey]

"That said, it may be the expectations or stereotype confound operates on UGPA, LSAT, law school grades, and the bar exam, thus making it hard to tease it out from existing data."

Well, I think that it is extremely likely that they operate at all of those points. Each successive criteria is measured and evolved to duplicate results on the former. (i.e. the LSAT is a "good" test because the same people that did well on it do well on law school exams.) There are institutional incentives and rewards when these things correlate. You have shown yourself how some criteria like "time pressure" can carry the correlation. It seems intuitively obvious that "implicit bias" or "stereotype threat" could do the same work in aligning various merit criteria relative to performance gaps.

I very much wanted to write an article discussing these linkages and entitled something like: "The Under-Theorized Impact of Systematic Implicit Bias in Replicating Peformance Gaps." I got so far as a presentation in Professor Lamber's discrimination seminar, but alas the requirement that I go to class and study limited my scholarly writing output. :) Now I have billable hours.

Jerry Kang wrote an article in... Harv. L. Rev. I believe called "The Trojan Horses of Race" where he summarized a lot of social science research on implicit bias and stereotype threat. For instance, apparently you can make Asian-Americans do better on a math test by reminding them that they are Asian before they start, and similarily can make women do worse by talking to them about feminism. I got a cold and that moved my practice LSAT score 10 points. It is a bogus test.

You don't have to tell me that the LSAT and the current hierarchy are here to stay. I have read Duncan Kennedy's books and noted that nothing has changed in the 25 years since they were written. This system is maintained in the same way as the caste system was for thousands of years. Nearly everyone is subordinate, but everyone has something to gain from asserting their own status over someone else below them. The biggest stakeholders in the concept of equality are the least empowered to assert their interests...

Better data is usually the answer but this particular data is inherently polarizing. Why? Because people are afraid that you will prove some inate difference between the races and cause a firestorm of controversy. (Witness the recent stupid comments of Dr. James Watson.) People don't want to believe that the testing system is biased and so a performance gap raises this tension. If they could only lose their faith in the test, then their faith in the fundamental equality of all races would not be challenged by a performance gap. The explanation would be simple, blame the test.

[Bill Henderson]

Corey,

These are good points. But the main problem is that virtually all validity studies on the bar (including the LSAC-BPS study by Wightman in 1998) have failed to find race to be a relevant variable for predicting bar passage. We would expect that outcome if one group has a lower performance because it is the target of lower expectations or stereotype. Right now, that data point is missing.

That said, it may be the expectations or stereotype confound operates on UGPA, LSAT, law school grades, and the bar exam, thus making it hard to tease it out from existing data. Frankly, no one has, to my knowledge, directly addressed this possibility. These are the types of rival hypotheses that can be explored if additional research can proceed forward with high quality data.

Regarding what the LSAT and Bar results mean, it is possible that they mean very little. Indeed, on this blog, I have pointed out that black students at Howard Law have excelled at Moot Court competitions -- i.e., simulating the actual practice of law, unlike law school exams, which are arguably less like practice -- and beaten schools like Harvard. This is a very interesting data point that warrants additional research. My earlier research has documented that testing method changes ordinal rankings. In simulated practice, perhaps many law professors would not fare very well even those most of us excelled on issuer spotter exams.

There are two issues here: (1) whether the existing credentialing process is sound and valid -- I am not sure that it is; and (2) taking the existing credentialing system as a given, what are the factors that cause minority candidates to underperform.

From an institutional perspective, scrapping the bar exam and the LSAT is a hard argument to make against large, entrenched actors. And legal educators and law school largely ignore the content of the bar exam as beneath them. It is not clear to me that calling for an abolition of these testing instruments is a strategy that will produce change. Exhausting all available research in point #1 and #2 above seems like the best tack to take. But you can't do that without good data. bh.

[Corey]

There are significant unquantifiable biases and subconscious discriminations that affect the experience of minority students in American law schools. I hope everyone reading this has taken the Implicit Association Test put online by Harvard (google it.) These influences are subtle but may be nevertheless quite systematic in their effect. Perhaps the largest of these biases is the persistent belief in mismatch theory. Imagine if everyone at your school believed you were mismatched. Knowledge of that preconception would ITSELF influence your ability to perform to your best. Some would be challenged and motivated by others doubts, but many more would be discouraged.

The fundamental output of Professor Sanders' research to date has been a received assertion that minority students subject to preferences are unqualified to perform at their schools. Against a background of persistent racial stereotyping that has NOT been eradicated, the mismatch theory gives people a logical justification for underestimating minority students. Anyone who has studied educational sociology knows that you can influence students' performance with your initial confidence (or lack thereof) in their ability to excel. Underestimating someone's prospects can limit their prospects.

So yes, this is an area where simply doing the research highlights a performance gap, and highlighting a performance gap causes real harm. If the data shows minority students underperforming, you don't have to understand why in order to motivate the types of positive programs you mention. Minority students would be helped by those programs whether their struggles were the result of "credential mismatch" or the result of persistent implicit bias, (made worse by increased focus on under-inclusive, biased, and performance-gapped "objective" credentialling.)

It may shock people that I am advocating discretion over the search for "truth." But researchers make choices about what to study every day. Both lawyers and empiricists often over-value "objective" criteria over subjective and empathic understanding. My assertion that you can make someone do better on a test simply by believing in them might seem silly to you all, but reality often seems absurd.

The fact that there is a minority performance gap should itself be sufficient to discredit the LSAT and provoke reforms at law schools. Otherwise you have to have a potentially career-ending conversation about racial differences in intelligence. Isn't it easier and more likely that the LSAT and law school classrooms are racially biased in ways we have not yet fully described and understood? I suggest that the focus be on implicit bias and not perfecting the predictive qualities of a suspect test.

[Dennis]

Your hypothesis sounds like a theory Thomas Sowell has been propounding for, seemingly, 30 years.

I don't know whether this hypothesis is right or wrong -- nor does anyone. Ed Henson above might be correct, or he might not be. But there's no way to know until the data are available.

But I suspect that people like Mr. Henson and others opposing the release of the data believe that there is a significant risk that they are wrong. Because if they truly believed that they were correct, they would have no reason to resist the release of the data -- the data would support their position, and the hypothesis would be rejected.

[Stuart Buck]

Note: Richard Lempert (Michigan Law and formerly chair of the NSF Law & Social Science Program), wrote this memo on the IRB issue, which he graciously forwarded to our research team in advance of the hearing

Correct me if I'm wrong, but isn't it the case that most of Prof. Lempert's memo does not address actual IRB issues, but rather presents his own reasons for thinking that "mismatch" research is misleading (or *could* be misleading)? In any event, it seems a bit alarming that one of the main disputants on this issue is so involved in attempting to justify the denial of data access to other researchers.

[cjt]

@ Ed Henson:

Your comments evince an attitude that is the intellectual equivalent of an ostrich sticking its head in the sand.

The Board of Governors ostensible reason(s) for declining to turn over the data is unfortunate (to put as good a face on it as possible).

[J. Kutherford]

"The whole notion of a so-called Mismatch Theory is simply outrageous."

So outrageous the establishment is too afraid to release the data that would allow the researchers to disprove it?

[Ed Henson]

Having attended the same hearing that professor Henderson writes about. It always boggles the mind. When one stipulates, prior to arguing a position to prove a negative. Well, I am a democrat, or a Liberal. My favorite one is I marched with the late Dr King. Therefore don't call me the R word. Professor Sander himself in his Bio lists his work in fair housing and having worked in the campaign of the late Harold Washington Black mayor of Chicago. It was no surprise that the Black Chairperson of the Civil Rights Commission, [Bush appointee]along with Joe Hicks former Los Angeles Human Relations Chair, and former director of the LA office of SCLC were in the group supporting Prof,Sander's "Mismatch theory ". Hey we're not biased look who's supporting us.Talking about well organized. It's no secret that some minority group members struggle with both the law school experience and the bar examination. Why ? primarily because of the socio/economic and cultural backgrounds of some of these students. You don't need to be a professor of law to understand that. When you examine the actual data that the State Bar of Ca does release as to Bar passage rates. It will show that all graduates, regardless of race or ethnicity of eg; Stanford and Boalt Hall two of the top tier law schools in Ca do well on the CA Bar. If the group in support of prof Sander is so intent on inproving law school success and bar passage rates of minorities. Then get involved in mentoring programs, involve yourself in efforts to improve early education programs in the minority communities. No matter how intellectually valid your arguments may appear to be. The whole notion of a so-called Mismatch Theory is simply outrageous.