Empirical Legal Studies

"False" Confessions, Plea Bargains, Experimental Designs & IRB

A recent paper now circulating on SSRN, The Innocent Defendant’s Dilemma: An Innovative Empirical Study of Plea Bargaining’s Innocence Problem, intrigues for an array of reasons. The subject matter--defendants' willingness to plead to offenses they did not commit incident to a plea bargain--certainly provokes. Also notable is the research design employed. The researchers, Lucian Dervan (S. Ill.) and Vanessa Edkins (psychology, Florida Tech.), constructed an experiment in which college students actually felt they were "accused of cheating and presented with her own incentives to admit wrongdoing and save the university the time and expense of proceeding before a disciplinary review board." The study involved "dozens of college students, and taking place over several months ... revealed that plea bargaining’s innocence problem is not isolated to an obscure and rare set of cases." Finally, insofar as this experiment involved human subjects, that the researchers managed to get their research project through IRB struck me as something of a minor miracle. An excerpted abstract follows.

"Strikingly, the authors’ study found that over half of the participants were willing to falsely admit guilt in return for a perceived benefit. This finding not only brings finality to the long-standing debate regarding the possible extent of plea bargaining’s innocence problem, but also ignites a fundamental constitutional question regarding an institution the Supreme Court reluctantly approved of in 1970 in return for an assurance it would not be used to induce innocent defendants to falsely admit guilt."

From the IRB: What to do if the Conversation Hurts You

Another IRB example for those who collect them:

“Because we conducted our research under the auspices of the IRB at Massachusetts General Hospital, we were required to use the hospital’s standard ‘informed consent’ form, which covers all of its highly diverse research—most of which involves drug trials or new medical and surgical procedures. Consequently, even though we were only talking to people in an office [about violent video games], we were obligated to tell the participants in our focus groups that if they suffered any side effects from our conversations, they should immediately go to the nearest hospital emergency room!”

Lawrence Kutner & Cheryl K. Olson, Grand Theft Childhood 67 (Simon & Schuster 2008).

Try To Get This Study Through Your University's IRB

Saw this in Time magazine and immediately thought about, among other things, challenges incident to research design and IRB.

IRB Approved?

I'm not sure if this research -- which is supported by the National Highway Traffic Safety Administration -- is subject to federal human subjects rules, but I wonder if any IRB would actually approve this method of recruiting participants:

The Gilpin County Sheriff's Office was apologizing Monday after a weekend effort to help a research group led to complaints about what appeared to be a DUI checkpoint - but wasn't.

Sheriff's officials who participated in the stops now acknowledge that the nonprofit organization requesting voluntary DUI and drug tests from drivers was overly persistent, according to complaints.

"It was like a telemarketer that you couldn't hang up on," said Gilpin County Undersheriff John Bayne.       

Sgt. Bob Enney said deputies assisted the Pacific Institute for Research and Evaluation in stopping motorists at five sites along Colorado 119 for surveys on any drug and alcohol use. Surveyors then asked the motorists to voluntarily submit to tests of their breath, blood and saliva. At least 200 drivers were tested, Enney said. About five motorists later complained, he said.

Roberto Sequeira, 51, said he and his wife, Terry, were detained for 15 minutes Friday evening despite their protestations that they needed to get their sleepy 10-year-old child back to their home in Nederland.

. . . .

Christopher N. Osher, Alcohol Surveys Spur Complaints, Denver Post, Sept. 18, 2007.

Empirical Legal Studies and IRBs

The Law & Society Association has created a committee to explore issues related to Institutional Review Boards (IRBs).  One of the committee members, William Gallagher (Golden Gate Law), asked that I post the following query.  I will post my own response in the comments section.

Malcolm Feeley, the current President of the Law and Society Association, has established a committee to examine issues relating to Institutional Review Boards (IRBs) and law and society research. As a committee member, I have been asked to look into potential issues that may affect law school-based scholars. As part of that effort, I am requesting legal scholars who do empirical work—broadly construed—in a law school setting to contact me with any information about their experiences with IRBs at their institutions. This is obviously not an attempt to do a comprehensive survey, simply to begin to focus on particular IRB issues that may pertain to law school researchers in particular (if any). 

Among the issues that I am interested in learning more about are (1) whether your law school has an IRB procedure that is separate from the rest of the university (if you are not in an independent law school); (2) whether your independent law school has an IRB; and (3) generally what your experiences have been with the IRB procedure at your institution—e.g., have you requested approval? A waiver? What result? Any additional information would be welcome as well.

If you are willing to provide this type of information, please send me a message by e-mail to: [email protected] [and/or post a comment below.  BH].

NYT IRB Story

Thought readers may be interested in this story in today's New York Times given our previous coverage of the issue.  (Thanks for the correction, Mike.)

More Perspectives on IRBs

Those interested in a slightly more sanguine perspective on the role of IRBs (especially in the bio-medical setting) will find Jerry Menikoff's (Kansas, Medicine) recent paper helpful (forthcoming in a Northwestern L. Rev. symposium issue).  An excerpted abstract follows:

"... Admittedly, the current system is imperfect and a variety of improvements can and should be made to eliminate those review requirements that do little to protect subjects. But even with these problems, getting IRB approval of social and behavioral research studies should in the great majority of cases be a relatively non-burdensome task that is a minimal hindrance to the conduct of the research. That burden should rarely if ever rise to the level of triggering constitutional protections against censorship."

More on IRBs

Though not itself empirical, David Hyman's (Illinois) recent piece (Institutional Review Boards: Is This The Least Worst We Can Do?, forthcoming in the Northwestern Univ. L. Rev.) will interest and inform anyone who has had the "pleasure" of dealing with an IRB incident to an empirical project.  David provides important perspective and context on the issues as well as helpful suggestions to improve the status quo.

IRB Mission Creep at Brown University

Ross Cheit recently posted a comment about his ongoing frustrations with the IRB process at Brown University. Since his comment is attached to an older post from our blog forum on IRBs, it may be missed by many readers. He said he wouldn't mind if I reproduced it here:

I have been fighting a losing battle against mission creep and other IRB craziness at Brown for years.

I have a collection of experiences that are almost too bizarre to believe. They prevented an undergraduate of mine from asking a question about views on reparations in a survey of other undergraduates because the question was considered "too sensitive." (She was told it had to go to full board review, and of course there was no time for that given that they had already spent months reviewing her thesis protocol.)

And they have just announced a new policy that says that undergraduates cannot be PIs on their own research. The reasoning: they are still learning the IRB process. We have tried without success to explain that they are also still learning to do research in general, which is why their work does not "contribute to generalizable knowledge"!

But no. Any opportunity to expand their jurisdiction is justified in the name of "ethics." It's madness. There is no sense that the federal statute never anticipated any of this. Meanwhile, they are expanding the definition of what undergraduate projects count as research.

I am discouraged and outraged, but at least because of this blog I no longer feel alone in this battle.

The original post by Jack Katz, "Outside of Biomed Research, IRBs are Essentially Censorship Agencies," is here.

Northwestern IRB Conference

Yesterday’s IRB conference at Northwestern University was well worth it.  In addition to the great panel discussions, the conference provided an opportunity to speak informally with IRB officials, who can be quite reasonable, pragmatic, and open to changing the current system to eliminate wasteful paperwork and delays.   Researchers and IRB officials can find common ground, but they need to be talking to one another at events like this conference to find it.  IRB “maximalists” accept the following proposition:  “No one should participate in research unless independent review concludes that the risks are reasonable in relation to the potential benefits.”  Nat’l Bioethics Advisory Commission, I Ethical and Policy Issues in Research Involving Human Participants ii (August 2001).  The people I spoke with were not so paternalistic and dogmatic.

Thanks to Philip Hamburger and James Lindgren for organizing the conference.

Coping with the “Petty Tyranny” of IRBs

Edward Abbey quipped that “No tyranny is so irksome as petty tyranny.” But, rather than fighting or railing against the system, my final post this week invites discussion on coping with IRBs. My prior post, and Jack Katz’s, talked about claiming exemption and when it’s safe to not even file. This post focuses on the “minimal risk” threshold. Below it, protocols may qualify for expedited review and waiver of formal consent, but beware ye who pass above it.

Richard Campbell, member of a now-defunct advisory committee, offers this excellent analysis: the Common Rule defines minimal risk as “not greater than those ordinarily encountered in daily life.” This should mean the sort and magnitude one encounters working, shopping, driving, or having a doctor’s check-up, which includes temporary or low-grade harms such as stress, anger, embarrassment, stubbed toes, paper cuts, etc. Campbell notes that risk of even disastrous harm meets the “everyday” standard if it’s fairly rare, such as an auto accident.

Building on the transportation example, Bill Ford articulated the absurdities of regulating car driving risks as we do research risks. It’s easy enough to point out differences between the two that justify regulating one but not the other. But the key point is to match the type of regulation with the degree of risk. Ford’s absurd example of filing route plans before driving is in fact how general aviation is regulated by the FAA (more or less) - which makes sense for its degree of risk. Similarly, the Common Rule, as written, could limit and modulate IRB oversight appropriately - if only IRB administrators could be convinced. In response to Ron Wright, perhaps that’s what law professors should be trying to do. 

Putting Lawyers into the IRB Mix

Most legal academics at most law schools have never submitted research proposals to an IRB. Most, I would guess, have no idea that their research probably falls within the asserted jurisdiction of the IRB on their campus.

If this benign (?) neglect disappears over the next few years -- as I think it will at many universities -- what will be the effects on practices in law schools, and what will be the effects on IRB practices? I have a prediction on the second question.

Based on other interactions between the university administration and the law faculty, I'm betting on some healthy turmoil. When the pension fund administrators at the university make claims and suggestions about personal and university investments, the law faculty start talking about securities fraud. When the university administration wants to "manage" the message coming from the college radio station, the First Amendment mavens at the law school get lots of air time.

The point is that law faculties, as a whole, have a reputation on many campuses for pushing back against assertions of administrative authority. They are feisty, even contentious. Other groups of faculty might prove just as feisty, but the law faculty neutralize one favored administrative technique -- claims about the law -- that work as debate-stoppers when it comes to other faculty groups.

I think this dynamic will hold true to form when law faculty start interacting with IRBs in larger numbers. When an IRB makes an overly broad assertion of power, based on its reading of the regulations, a lawyer is more likely than a biologist or a sociologist to reject the current way of doing business and propose something different. If, as Jack Katz mentioned in his recent post, an IRB sees its mission as preventing research that could provoke some legal claim against the university, a legal researcher will be pretty vocal in pointing out the weaknesses of such a civil claim. And what does prevention of long-shot civil claims have to do with the Belmont principles?

If you chaired an IRB, wouldn't you quietly hope that legal academics remain in a gray area of possible non-compliance, just for the peace and quiet? 

Outside of Biomed Research, IRBs are Essentially Censorship Agencies

Like many ethnographers, I often do my research by putting myself in places and relating to people without a very clear idea as to what I’m looking for.  Even when I have a clear idea, like when I’m going to interview someone about their life in a neighborhood, I’ll see things on the way there and back that may be more valuable than what I find out when my questions get answered.  Much of my fieldwork becomes “data” in retrospect.  This makes it impossible to clear much of my research beforehand with an IRB.

To different degrees, many social researchers, not just anthropologists but political scientists and many others who would create “generalizable knowledge” and thus fall within IRB jurisdiction, work this way.  So do law professors, as entries on this blog have already acknowledged.

This means we are all vulnerable to attack for violating the rules in our universities or colleges. There are ways of limiting our vulnerability; I’ve detailed some in a paper that’s part of a collection currently being considered by an anthro journal.

Studying IRBs - Goldman, Katz, and Smith

Jerry Goldman and Martin Katz published a rare empirical study of IRBs in the Journal of the American Medical Association in 1982.  Here is a brief summary:

We submitted three flawed medical protocols with the expectations that the review boards would find the flaws evident in the proposals. But the boards, on the whole, did not find the flaws. Rather, they suggested changes that would, if approved, made bad research projects worse. This is the only major empirical study of IRBs and the findings have never been controverted with other evidence.

The paper can be found here.  Professor Goldman may have already started a replication of this study with Dr. Lewis Smith. (Thanks to Steve Wasby for calling my attention to this paper.)

IRBs and Regulatory Theory

When I first saw an IRB at work, it reminded me immediately of a creature that is familiar to Administrative Law teachers: the agency that has incentives to ignore Type 2 errors in favor of Type 1 errors. The classic example is the Food and Drug Administration. The FDA knows that if it wrongly approves a dangerous drug for sale (Type 1 error), then when somebody dies after using the drug (say, using thalidomide as a sleep remedy for pregnant women), there is hell to pay. Public outrage, congressional hearings, you know the drill. On the other hand, if the FDA wrongly fails to approve -- or wrongly delays approval -- of a safe and useful drug (Type 2 error), the costs are more diffuse and less visible. There are exceptions (think of recent pressure on the FDA over AIDS treatment drugs), but the culture of the FDA pushes everyone in the agency to be most alert for Type 1 errors, even though both types of errors are costly to health.

The same incentives operate within a university IRB. The consequences of Type 1 errors -- such as wrongly approving harmful or non-complying research protocols -- are disastrous, including suspension or loss of funding for the entire institution. The IRB, on the other hand, tends to ignore the consequences of Type 2 errors, such as the delays in research, the research projects that never occur, the research that is needlessly scaled back in ambition.

This old chestnut from regulatory theory leaves a sobering message about IRB reform. It is not enough simply to point out the possible Type 2 errors and expect the IRB to account for them from now on. There are powerful reasons why Type 1 errors get the most attention. The strategies have to focus on ways to make Type 2 errors more visible and salient across many different bureaucratic cultures and across research proposals from many different parts of the academic world.

What we need are ideas that are universal amplifiers for Type 2 errors, loud enough to make stifling of research a routine worry for the IRB, on an equal footing with the Type 1 errors. Got any?