For those readers who are unfamiliar with Moneyball, it refers to the use of statistical methods to identify and exploit inefficiencies in the market for baseball talent. In a widely read review essay on its potential implications for law schools, Paul Caron and Rafael Gely assembled statistical evidence showing that the hiring heuristics used by most law schools (e.g., law school attended, fancy clerkship) were poor predictors of quantity/placement of future scholarship. Rather, the best predictors of future productivity were pre-academy productivity and the publication of a student note.
Aside from Caron & Gely's preliminary work, however, the collective ruminations of the Moneylaw contributors have not (yet, anyway) articulated a theoretically coherent basis for how Moneyball/Moneylaw principles can produce a more successful law school franchise. (Jim, Jeff, Ted, Paul, Nancy, Al, I say this with love in my heart.) Since we don't have wins and losses or a World Series title, the most obvious theoretical shortcoming is how success is measured, both in the short and long term. (In an earlier post, I suggested money as the best longterm metric.) Once this theoretical work is in place, empirical methods can be brought to bear to generate the appropriate chess moves.
So I was quite surprised to receive a phone call the other day from an administrator at a non-Tier 1 school whose primary charge is identifying data-driven ways to improve the functioning of the school. For the past several years, the law school has paid for this administrator to acquire sophisticated statistical training and build the requisite datasets. (Jim Chen, you have some serious competition!) The approach is pretty simple: pick an outcome that matters and generate an inductive theory through data mining. This "methodology" is not very academic, but it is how Moneyball was invented; it also reflects the approach of many successful hedge funds.
So what outcomes matter? Here are a few suggestions, some borrowed from the Moneyball/Moneylaw administrator:
- Bar passage (LSSSE adds a whole new layer of variables), including factors that reduce or eliminate any minority passage gap
- Employment at graduation / Employed at 9 months (regressions published here)
- Credentials of incoming students (regressions published here)
- Annual giving (rate, average amount)
- Career satisfaction of graduates
- Satisfaction of major employers of graduates
- Faculty participation in law reform / participation in successful law reform
- Pro bono commitment of graduates
If any readers have any more, I will post them. Each item above requires brainstorming of possible independent variables and the appropriate modeling technique. In addition, it would be worthwhile to add some empirical content to the following questions:
- What is the institutional payoff for scholarship? Does it measurably increase law school prestige, which in turn affects enrollment or annual giving? If so, how large is the effect? Are gains limited to a particular market segment? Alternatively, are all institutional gains dissipated by lateral movement?
- Do teaching evaluations provide any information that could be used to improve any law school outcome (e.g., bar passage, employment, eventual annual giving, career satisfaction)? Do any of these questions turn on the content or timing of the survey instrument? (See Ben Barton's study and the subsequent ELS discussion.)
- Does any curricular initiative, such as a clinic or certificate program, produce higher employment rates, higher earnings, higher graduate satisfaction, or higher employer satisfaction?
I am sure many ELS and Moneylaw readers can offer significant improvement to this preliminary list.