Empirical Legal Studies

Helpful Reminders on How To Minimize Errors in Empirical Research

Responding to the recent debacle involving a grad student uncovering a blundering error in a paper by noted Harvard economists (here), Betsey Stevenson (Mich.) and Justin Wolfers (Mich.) initiated a (now growing) list of suggestions on how to minimize errors in empirical research. Not surprisingly, others, including Andrew Gelman (Columbia), have added to the list (here). While the list will inevitably grow, it already includes basic, helpful reminders for even the most experienced researchers.

(Re-)Defined Data

Over at PrawfsBlawg, John Pfaff (Fordham) provides a cautionary reminder that most empiricists cannot hear often enough. Re-defining key variables can influence results and re-defined variables are frequently difficult to detect, particularly in large, complex, longitudinal datasets. That is, too often, secondary analyses are undertaken without necessary due-diligence involving the underlying data. Among Pfaff's take-aways:

"If nothing else, this is a strong warning against casually running empirical models, a growing problem in legal scholarship. Legal academics shouldn’t just get their IT departments to install Stata on their computers, download some data, and then start running some regressions. It  can take years to fully understand what a dataset looks like, what it is really measuring, its strengths and weaknesses. People who just run some quick regressions and then send them off to a law review are likely moving knowledge backwards, not forwards, since the risk of bad results is too great."

Emerging Data on Stata's YouTube Channel

While noting that Stata does not shy away from holding its own initiatives up to data should not surprise, a summary report on the initial six months of Stata's YouTube Channel usage might interest readers. If there is any surprise in a summary (and descriptive) blog post (here), perhaps it's the degree to which this project is succeeding. What Stata is doing is certainly generating interest (thus far, anyway) and I assume we'll see similar efforts by others in the near future.

Stata and Multilevel Modeling

A pair of recent posts (and related videos) on the Stata Blog involving multilevel modeling warrants note. Part 1 (and related video) and Part 2 (and video) walk readers through such topics as hierarchical or multilevel data as well as how to analyze longitudinal data with Stata’s xtmixed command. Insofar as many empirical legal projects involve data structurally nested within counties and/or states, circuits, etc., multilevel techniques are quite important.

2013 Causal Inference Workshops: Main and Advanced

Information on two separate (Main and Advanced) Causal Inference Workshops at Northwestern Law School this summer follows. Both workshops will be taught by world-class causal inference researchers. Registration for each is limited to 100 participants.

  

Main workshop: Monday – Friday, June 24-28, 2013 

Advanced workshop: Monday - Wednesday, August 12-14, 2013.

For information and to register for either or both workshops: (click here)

Main Workshop Overview and Target Audience: Most empirical methods courses survey a variety of methods. We will begin instead with the goal of causal inference, and discuss how to design research to come closer to that goal. The methods are often adapted to a particular study. Some of the methods we will discuss are covered in PhD programs, but rarely in depth, and rarely with a focus on causal inference and on which methods to prefer for messy, real-world datasets with limited sample sizes. Each day will include with a Stata “workshop” to illustrate selected methods with real data and Stata code. We will assume knowledge, at the level of an upper-level college econometrics or similar course, of multivariate regression, including OLS, logit, and probit; basic probability and statistics including conditional and compound probabilities, confidence intervals, t-statistics, and standard errors; and some understanding of instrumental variables.

Advanced Workshop Overview and Target Audience: The advanced workshop seeks to provide an in-depth discussion of selected topics at the causal inference research frontier. Our target audience is empirical researchers who are familiar with the basics of causal inference (from our main workshop or otherwise), and want to extend their knowledge.

Conducting Empirical Legal Scholarship Workshop, May 22-24

The Conducting Empirical Legal Scholarship Workshop is for law professors and other members of the legal community, social science faculty, and graduate students interested in learning about empirical research.  Leading empirical scholars Lee Epstein and Andrew Martin teach the workshop and provide the formal training necessary to design, conduct, and assess empirical studies and use statistical software (Stata) to analyze and manage data.

 

Participants need no background or knowledge of statistics to enroll in the workshop.

 

Register Now!

https://mylaw.usc.edu/empiricalWorkshop/registerOnline.cfm

 

May 22 - 24, 2013

 

The 12th Annual Conducting Empirical Legal Scholarship Workshop 

at the University of Southern California Gould School of Law

Presented by USC Gould School of Law and Washington University Law

 

 

For more information, visit:  http://law.usc.edu/EmpiricalWorkshop 

 

Instructor  

Lee Epstein, Provost Professor and Rader Family Trustee Chair in Law and Political Science at University of Southern California, is a leading empirical legal scholar and a Fellow of the American Academy of Political and Social Science and American Academy of Arts and Sciences. She has co-organized and co-led this annual empirical scholarship workshop for the past eleven years.

 

Instructor  

Andrew Martin, Professor of Law and Political Science, and Director of the Center for Empirical Research in the Law at Washington University, specializes in political methodology and has written widely on American political institutions, including the Supreme Court and the Courts of Appeals. He has co-organized and co-taught the empirical scholarship workshop with Professor Epstein for the last eleven years.

 

2013 Main Workshop on Research Design for Causal Inference

Information on the 4th annual workshop, organized by Bernie Black (Northwestern) and Mat McCubbins (USC), follows. This year's workshop is scheduled for June 24-28, 2013 at Northwestern University Law School. As the workshop enrollment is capped at 100, those interested are encouraged to register sooner rather than later.

* * *

We would like to invite you to attend the fourth (annual) workshop on Research Design for Causal Inference, sponsored by Northwestern University, the University of Southern California, and the Society for Empirical Legal Studies. We have recruited a world-class faculty for the workshop – see below for details.

Bernie Black [Northwestern, Law School and Kellogg School of Management]

Mat McCubbins [USC, Law School, Marshall School of Business, and Political Science Department]

Dates and Location: Monday – Friday, June 24-28, 2013, at Northwestern Law School, Chicago, Illinois.

Registration: information is available here. Registration is limited to 100 participants. We filled up quickly last year, so please register soon. See below for cost and other details.

Overview of the Causal Inference Workshop: Research design for causal inference is at the heart of a “credibility revolution” in empirical research. We will cover the design of true randomized experiments and contrast them to simulations and quasi-experiments, where part of the sample is “treated” in some way, and the remainder is a control group, but the researcher controls neither the assignment of cases to treatment and control groups nor administration of the treatment. We will assess the kinds of causal inferences one can and cannot draw from a research design, threats to valid inference, and research designs that can mitigate those threats.

Most empirical methods courses begin with the methods. They survey how each method works, and what assumptions each relies on. We will begin instead with the goal of causal inference, and discuss how to design research to come closer to that goal. The methods reflect the goal and are often adapted to the needs of a particular study. Some of the methods we will discuss are covered in PhD programs, but rarely in depth, and rarely with a focus on causal inference and on which methods to prefer for messy, real-world datasets with limited sample sizes.

Each day will conclude with a Stata “workshop” where we will illustrate selected methods with real data and Stata code.

Target audience: Quantitative empirical researchers (faculty and graduate students)in social science, including law, political science, economics, many business-school areas (finance, accounting, management, marketing, etc), medicine, sociology, education, psychology, etc. – indeed anywhere that causal inference is important.

Minimum prior knowledge: We will assume knowledge, at the level of an upper-level college econometrics or applied statistics course,of how to run multivariate regressions, including OLS, logit, and probit; familiarity with basic probability and statistics including conditional and compound probabilities, confidence intervals, t-statistics, and standard errors; and some understanding of instrumental variables.

Despite its modest prerequisites, this course should be suitable for most researchers with PhD level training and for empirical legal scholars with reasonable but more limited training. For recent PhD’s, there will be overlap with what you already know, but much that you don’t know, or don’t know as well as you should.

Registration and Workshop Cost

Tuition is $850; with a discounted rate of $500 for graduate students (PhD, SJD, or law) and post-doctoral fellows. The workshop fee includes all materials, a temporary Stata12 license, breakfast, lunch, snacks, and Monday evening reception. All amounts will increase by $50 on April 1, 2013 (but we’re likely to fill up well before then). Registration deadline: June 10, 2013. You can register online at the url above. You can cancel by May 3, 2013 for a 75% refund and by May 31, 2013 for a 50% refund (in each case, less credit card processing fee), but there are no refunds after that.

For Northwestern or USC-affiliated attendees, we will charge the regular rate, but will give you a refund after the workshop to bring your cost down to $300 if you in fact attend at least a majority of the sessions. We adopted this policy because if you register and don’t come, you took a spot we could have provided to someone else.

We know the workshop is not cheap. We use the funds to pay our speakers and for meals and other expenses; we don’t pay ourselves.

Questions about the workshop: Please email Bernie Black (bblack@northwestern.edu) or Mat McCubbins (mmccubbins@law.usc.edu) for substantive questions or fee waiver requests, and Michael Cooper (causalinference@law.northwestern.edu for logistics and registration questions.

Postestimation: Margins & Marginsplot in Stata

A helpful recent addition to the Stata archive of tutorial videos involves the "margins" and "marginsplot" postestimation commands. This particular video example illustrates how to graph predictions from a linear regression model with an interaction between continuous and categorical covariates.

Should We Expect Expectation Effects?

An interesting discussion recently emerged (here and here) of a study comparing two agricultural experiments (involving different seeds of cowpeas to farmers in Tanzania) -- one blinded, one unblinded. As Andrew Gelman (Columbia) notes: "Bulte et al. find much different results in the two experiments and attribute the difference to expectation effects (when people know they’re receiving an experiment they behave differently); Ozler is skeptical and attributes the different outcomes to various practical differences in implementation of the two experiments."

Modeling Long (and Triple) Jumping

Andrew Gelman (Columbia) passes along (and seeks input on) this intereting post from the long (and triple) jump coach at Boise State University, Jeff Petersmeyer (who also coached horizontal jumpers at the 2012 Olympics). Jeff wonders how to better deploy statistical modeling into the service of predicting jumpers' potential. Evidently, a "Moneyball" perspective is migrating into the world of track & field.