Empirical Legal Studies

Coping with the “Petty Tyranny” of IRBs

Edward Abbey quipped that “No tyranny is so irksome as petty tyranny.” But, rather than fighting or railing against the system, my final post this week invites discussion on coping with IRBs. My prior post, and Jack Katz’s, talked about claiming exemption and when it’s safe to not even file. This post focuses on the “minimal risk” threshold. Below it, protocols may qualify for expedited review and waiver of formal consent, but beware ye who pass above it.

Richard Campbell, member of a now-defunct advisory committee, offers this excellent analysis: the Common Rule defines minimal risk as “not greater than those ordinarily encountered in daily life.” This should mean the sort and magnitude one encounters working, shopping, driving, or having a doctor’s check-up, which includes temporary or low-grade harms such as stress, anger, embarrassment, stubbed toes, paper cuts, etc. Campbell notes that risk of even disastrous harm meets the “everyday” standard if it’s fairly rare, such as an auto accident.

Building on the transportation example, Bill Ford articulated the absurdities of regulating car driving risks as we do research risks. It’s easy enough to point out differences between the two that justify regulating one but not the other. But the key point is to match the type of regulation with the degree of risk. Ford’s absurd example of filing route plans before driving is in fact how general aviation is regulated by the FAA (more or less) - which makes sense for its degree of risk. Similarly, the Common Rule, as written, could limit and modulate IRB oversight appropriately - if only IRB administrators could be convinced. In response to Ron Wright, perhaps that’s what law professors should be trying to do. 

Putting Lawyers into the IRB Mix

Most legal academics at most law schools have never submitted research proposals to an IRB. Most, I would guess, have no idea that their research probably falls within the asserted jurisdiction of the IRB on their campus.

If this benign (?) neglect disappears over the next few years -- as I think it will at many universities -- what will be the effects on practices in law schools, and what will be the effects on IRB practices? I have a prediction on the second question.

Based on other interactions between the university administration and the law faculty, I'm betting on some healthy turmoil. When the pension fund administrators at the university make claims and suggestions about personal and university investments, the law faculty start talking about securities fraud. When the university administration wants to "manage" the message coming from the college radio station, the First Amendment mavens at the law school get lots of air time.

The point is that law faculties, as a whole, have a reputation on many campuses for pushing back against assertions of administrative authority. They are feisty, even contentious. Other groups of faculty might prove just as feisty, but the law faculty neutralize one favored administrative technique -- claims about the law -- that work as debate-stoppers when it comes to other faculty groups.

I think this dynamic will hold true to form when law faculty start interacting with IRBs in larger numbers. When an IRB makes an overly broad assertion of power, based on its reading of the regulations, a lawyer is more likely than a biologist or a sociologist to reject the current way of doing business and propose something different. If, as Jack Katz mentioned in his recent post, an IRB sees its mission as preventing research that could provoke some legal claim against the university, a legal researcher will be pretty vocal in pointing out the weaknesses of such a civil claim. And what does prevention of long-shot civil claims have to do with the Belmont principles?

If you chaired an IRB, wouldn't you quietly hope that legal academics remain in a gray area of possible non-compliance, just for the peace and quiet? 

Outside of Biomed Research, IRBs are Essentially Censorship Agencies

Like many ethnographers, I often do my research by putting myself in places and relating to people without a very clear idea as to what I’m looking for.  Even when I have a clear idea, like when I’m going to interview someone about their life in a neighborhood, I’ll see things on the way there and back that may be more valuable than what I find out when my questions get answered.  Much of my fieldwork becomes “data” in retrospect.  This makes it impossible to clear much of my research beforehand with an IRB.

To different degrees, many social researchers, not just anthropologists but political scientists and many others who would create “generalizable knowledge” and thus fall within IRB jurisdiction, work this way.  So do law professors, as entries on this blog have already acknowledged.

This means we are all vulnerable to attack for violating the rules in our universities or colleges. There are ways of limiting our vulnerability; I’ve detailed some in a paper that’s part of a collection currently being considered by an anthro journal.

Studying IRBs - Goldman, Katz, and Smith

Jerry Goldman and Martin Katz published a rare empirical study of IRBs in the Journal of the American Medical Association in 1982.  Here is a brief summary:

We submitted three flawed medical protocols with the expectations that the review boards would find the flaws evident in the proposals. But the boards, on the whole, did not find the flaws. Rather, they suggested changes that would, if approved, made bad research projects worse. This is the only major empirical study of IRBs and the findings have never been controverted with other evidence.

The paper can be found here.  Professor Goldman may have already started a replication of this study with Dr. Lewis Smith. (Thanks to Steve Wasby for calling my attention to this paper.)

IRBs and Regulatory Theory

When I first saw an IRB at work, it reminded me immediately of a creature that is familiar to Administrative Law teachers: the agency that has incentives to ignore Type 2 errors in favor of Type 1 errors. The classic example is the Food and Drug Administration. The FDA knows that if it wrongly approves a dangerous drug for sale (Type 1 error), then when somebody dies after using the drug (say, using thalidomide as a sleep remedy for pregnant women), there is hell to pay. Public outrage, congressional hearings, you know the drill. On the other hand, if the FDA wrongly fails to approve -- or wrongly delays approval -- of a safe and useful drug (Type 2 error), the costs are more diffuse and less visible. There are exceptions (think of recent pressure on the FDA over AIDS treatment drugs), but the culture of the FDA pushes everyone in the agency to be most alert for Type 1 errors, even though both types of errors are costly to health.

The same incentives operate within a university IRB. The consequences of Type 1 errors -- such as wrongly approving harmful or non-complying research protocols -- are disastrous, including suspension or loss of funding for the entire institution. The IRB, on the other hand, tends to ignore the consequences of Type 2 errors, such as the delays in research, the research projects that never occur, the research that is needlessly scaled back in ambition.

This old chestnut from regulatory theory leaves a sobering message about IRB reform. It is not enough simply to point out the possible Type 2 errors and expect the IRB to account for them from now on. There are powerful reasons why Type 1 errors get the most attention. The strategies have to focus on ways to make Type 2 errors more visible and salient across many different bureaucratic cultures and across research proposals from many different parts of the academic world.

What we need are ideas that are universal amplifiers for Type 2 errors, loud enough to make stifling of research a routine worry for the IRB, on an equal footing with the Type 1 errors. Got any? 

Confessions of an Undocumented Human Subjects Researcher

I confess.  Like Jimmy Carter, I have lusted in my heart, and like most law professors, I have engaged in unprotected human subjects research (HSR).  Just last week, for instance, I called a lawyer and an insurance executive to get their views on a legal issue I was researching, and I might cite them in the article I’m writing.  Later, I wondered whether I should have had IRB permission first to do that. 

I expect that a good portion of this forum on IRBs will address overregulation, but I doubt we’ll be able to do much about that in the short term, so I would like to kick off a second line of discussion this week: practical tips for coping with IRB oversight.  To focus discussion, note that there are two regulatory thresholds: 1) is an activity non-exempt human subjects research (HSR) to begin with; and 2) if so, does it pose more than minimal risk?  This post focuses on the first of these two thresholds. 

As summarized in the IRB FAQs that Ron Wright and I posted several weeks ago, there are several bases to claim that interaction with, or data about, living humans in the course of legal scholarship is not a covered activity (not identifiable, not systematic, not private, etc.).  Naturally, those concepts have room for interpretation.  The hitch is that institutions expect researchers to notify the IRB so it can have the final say on whether the exclusionary concepts apply. But, that quickly leads to innumerable absurdities.  Every hour of every working day, I do things that are not HSR.  Which of those things do I need to tell the IRB about, simply to confirm they aren’t non-exempt HSR?  Do we also need to define a grey zone threshold for when an activity MIGHT be interpreted as covered HSR by an IRB?

Not necessarily.  I believe this threshold decision is one that researchers should be able to make on their own in good faith, albeit at their own risk of being wrong.  If I’m not seeking external funding, and I’m not publishing the research in journals that routinely deal with HSR, then it’s not likely that anyone will require me to verify the absence of HSR.  And, if in fact I haven’t done any HSR, then have I done anything wrong by not clearing the activity first with the IRB?   If not, then maybe I only need to seek IRB approval if I’m not sure about the activity being HSR, OR if I anticipate having to verify to an outsider (journal or funding agency) that it isn’t covered HSR.  Am I wrong about this?  How much trouble could I or the institution get into for doing things that aren’t covered HSR without seeking IRB approval first?

Addendum:  Jack Katz’s related post refers to an excellent paper of his (available soon here, and in press at the American Ethnologist), that develops two key points in detail:  1) IRBs are not required to review research that is not federally funded.  Although most Universities require this, some prominent ones are starting to step back from that.  2) Some types of research involving humans are not sufficiently “systematic” to meet the Common Rule’s threshold definition.  It’s a really good paper; you should read it. 

Human Research Protection Programs (HRPP) Blog

The HRPP Blog offers a perspective on IRBs from someone with extensive experience as an IRB professional and consultant.  There are only a small number of posts so far, but he does comment on the Center for Advanced Study's "Mission Creep" white paper in the most recent one.

Please Have Your Consent Forms Ready

Is change afoot in the regulation of social science research by IRBs, especially low risk research involving surveys, interviews, and the use of already existing data?  One source of change could be the intensifying scrutiny of IRBs. In April 2003, the Center for Advanced Study held a conference on IRBs and published a white paper on the topic, Improving the System for Protecting Human Subjects: Counteracting IRB “Mission Creep.”  Next month at Northwestern University, Philip Hamburger (Columbia) and James Lindgren (Northwestern) are holding a conference on the First Amendment implications of IRB regulations.  If academic conferences can provoke change anywhere, maybe it’s in academia.  Another possible source of change is the increasing interest of legal scholars in empirical research, which means more IRB oversight of their work.  Presumably, regulating legal scholars increases the likelihood of an unhappy researcher filing a lawsuit against an IRB -- and perhaps making the arguments generated at these conferences to a court.  Whether changes are likely and what changes are desirable are two of the topics up for grabs this week.

During this second Blog Forum, four guest bloggers will offer their perspectives on IRBs.  Kristina Gunsalus of the University of Illinois’ Office of University Counsel and the College of Law (Urbana-Champaign) has written widely on the topic of IRBs.  She is the primary author of the Center for Advanced Study’s white paper.  Mark Hall and Ron Wright of Wake Forest University’s School of Law have experience from the law school side of dealing with IRBs.  Professor Hall specializes in health care law and policy.  Professor Wright specializes in criminal justice and administrative law.  Jack Katz of U.C.L.A.’s Sociology Department is a participant in the upcoming conference at Northwestern University.  He specializes in social psychology; ethnographic methods; urban communities; and crime, law and deviance.

Next Week: A Blog Forum on Institutional Review Boards (IRBs)

Suppose long ago the local Departments of Motor Vehicles, perhaps under the auspices of the Federal Motor Carrier Safety Administration, started regulating the transportation of hazardous chemicals on public roads. They would inquire about destinations, routes, and safety protocols. Since the risks from these materials may be significant, this oversight might be a good idea. But at some point, assume they decided to monitor pretty much all travel on the public roads: No one should travel anywhere unless independent review concludes that the risks are reasonable in relation to the potential benefits. Cf. Nat’l Bioethics Advisory Commission, I Ethical and Policy Issues in Research Involving Human Participants ii (August 2001). In other words, no one should be judging whether to go anywhere for him or herself, until after someone else has decided it’s okay. See C. K. Gunsalus, The Nanny State Meets the Inner Lawyer: Overregulating While Underprotecting Human Participants in Research, 14 Ethics & Behavior 369-382 (2004).

At least for now in this hypothetical world (FN1), travel plans on private roads are not covered at all, and many travel plans on public roads are exempt from the DMV’s oversight. If it involves travel on public roads, however, only the DMV can judge it as exempt. Some common plans are pre-approved as exempt, e.g., travel to local grocery stores along certain routes. But you probably still need to send an e-mail to your DMV notifying them of your intent to use these pre-approved plans. They’d like to know what you’re up to.