Combining Quantitative and Qualitative Methods in Socio-legal Research

In their opening comments and responses to other posts, Beth Mertz and Stewart Macaulay ably made the connection between the New Legal Realism (NLR) and established traditions of empirical inquiry among law and society scholars.  Stewart reasserted a theme of his own classic work, that legal scholars should seek "a more accurate picture of law in action" before they offer proposals for changing law.  Beth called attention to the need for socio-legal scholars to more self-consciously translate their research findings to have a broader effect within the legal academy, among policy audiences, and among the public.   Both  advocated  a multiplicity of  empirical methods,  a self-critical humility for the nature of the claims that can be supported by different modes of empirical research, and at least an implicit recognition of the need to connect empirical inquiry to broader theoretical propositions about law and society.  As empirical legal studies expands and matures in the legal academy, these principles are important for the intellectual significance of the enterprise. 

In this essay I want to briefly argue that theoretically driven research that uses multiple methods can produce stronger validity claims, can better illuminate the social mechanisms through which law operates, and may lead to research findings that more readily translate to broader publics.  In other words, research that advances the goals of NLR.  I will try to carry the argument with the examples of three research projects, one by a colleague and two in which I have been personally involved.

Laura Beth Nielsen's License to Harass (Princeton University Press 2004) studied offensive public speech through observation and in-depth interviews with some 100 respondents, including white men, white women, and men and women of color.  She found that the experience of these groups with sexist and racist speech on the street varied dramatically by social group, with women and people of color encountering harassment at shocking levels.   While all social groups  saw  offensive speech as a serious social problem,  a large majority of respondents in all social groups opposed legal intervention to deal with it.  But the groups based their positions on different rationales.  White men articulated a classic first amendment view, women invoked a discourse of autonomy (they would take care of it themselves), and people of color (especially men)  spoke of a distrust of authority (they thought the law would be used against them).   Without an in-depth investigation of the discourses employed by these different groups, we would know much less about their legal consciousness concerning offensive public speech.  Nielsen is thus able to offer some profound insights into the experiences of different social groups on the street, their attitudes about the law, and about the role of law in shaping those attitudes and providing a license for offensive speech to continue to operate.

In Legalizing Gender Inequality  (Cambridge University Press 1999), William Bridges and I examined four case studies of litigation concerning gender-based pay discrimination.  Because the cases involved comparisons of wages across different predominantly male and predominantly female jobs, the cases were characterized as comparable worth cases.  The defendant employers in these cases invoked a market defense, arguing that the wage differentials reflected market processes rather than gender discrimination.  The courts in 3 of the 4 cases accepted that defense.  Our book subjects those legal holdings to empirical scrutiny through a variety of labor market, employment payroll data, organization process data, and in-depth interviews.   The different forms of data converge to reject the market defense.  For example, in AFSCME v. State of Washington,   we found that a state pay survey that was treated as a neutral scientific instrument for determining job-level pay was in fact riddled with arbitrariness and bureaucratic politics.  As a state labor union representative commented, they were not "going to rob Peter to pay Pauline."   The book concludes that the courts in these cases  interpreted antidiscrimination law as  potentially providing a remedy for between-job pay discrimination,  while at the same time refusing to critically analyze actual wage-setting practices in these organizations.  The result was  that the courts legitimated between-job gender inequality in pay.

The final example concerns a project currently underway by Laura Beth Nielsen, John Donohue, Peter Siegelman, and myself.  The project examines the changing dynamics of employment discrimination litigation from 1987 to 2002.  We possess a full set of EEOC complaints from that period, in addition to a large random sample of federal court filings from 7 jurisdictions, which we have coded and begun to analyze quantitatively.  The statistical data will allow us to comprehensively  examine  the system  of discrimination disputing and to test whether it is operating differently than  during the 1972-1987  period analyzed by Donohue and Siegelman in their earlier work.    Employment discrimination litigation has become an intensely contested arena.  Many commentators assert that the dramatic rise in discrimination lawsuits, which roughly tripled between 1991 and 1998,  is a result of growing numbers of frivolous lawsuits.   

We are  also interviewing in depth parties and their lawyers  in a selected subset of  cases.   After conducting some 50 in-depth interviews thus far, we already have gained important insights into how the system operates.  The interviews have literally brought these cases to life.   One African-American plaintiff described how he had been asked by his immediate supervisor whether he had ever had sex with his daughters, as he had heard that slaves used to do.  When internal complaints did not remedy his work environment, he filed suit.  In our dataset he is coded as an early settlement on the outcome variable.  But what the coded file does not indicate is the gravity of the insult he received, the fact that as a result of the lawsuit he became divorced, went bankrupt, felt enormous physical stress.  Although he won his job back as part of the settlement, he was laid off in a reduction in force a year later.  When asked whether he would file the lawsuit again, he said no, that he "would have took it."   In this case, as with almost all the plaintiffs we have interviewed, we find serious injuries, even though the proof of illegal discrimination sometimes is shaky.  Indeed, the defense attorneys largely concede that most of the cases they encounter involve individuals who have been "wounded" at work. 

While we have more data to collect and analyze, the combination of systematic statistics and in-depth interviews is likely to produce a better understanding of antidiscrimination law in action than presently exists.   This combination should allow us to disseminate our results more effectively to legal, academic, policy, and public audiences than might be the case with only qualitative or quantitative data.   

Comments

[Jay mangal prasad]

Please find full description about census in legal research.

[Laura Beth Nielsen]

Whatever we call ourselves -- New Legal Realists, Empirical Legal Studies or (as I think of myself) good old fashioned Sociolegal Studies/Law and Society (just for fun we could revisit the debate about Law & Society vs. Sociolegal Studies vs. Jurisprudence and Social Policy vs. Law and Social Science) -- method is important. I daresay, WAY more important than the label(s).

I think it would be unfortunate in this field to reproduce the hierarchies of methodology that have plagued other disciplines like sociology and political science. It does not help us move forward in how we understand law and its relationship to society/social problems.

As most people here have said -- it is about the fit of the data to the question being asked (and, of course, how interesting and important the question is).

Multi-method is being embraced in sociology right now as a possible way to get over this great divide and to better answer the questions. It does require expertise on various methods -- mostly we train in one or two. That's why Bob Nelson, me, John Donohue, and Peter Seigelman have teamed up -- I'm the qualitative one, John and Peter have the deep stats basis. Bob has always done a bit of both. I'm learning tons from them but would not be comfortable with my ability to do much of the quantitative work without collaborators. But, I would not be confident in their work without seriosuly thinking through how we want to create these models without my input about different theoretical reasons for including and excluding different variables. We need each other to get the questions and answers right.

[Elizabeth Mertz]

Bill, I appreciate your suggestions for opening up further discussion on this. Of course any such discussion should include the Law & Society Review as well --

[William Henderson]

Bob (and Beth and Stewart),

This is a great thread that is reminiscent of Howard Gillman's provocative guest post a few weeks ago when he talked about empirical methodology (quantitative and qualitative being the two main poles) being a function of the question being asked. Every time the quantitative versus qualitative (or "nonquantitative" as one astute commenter has called it) debate comes up, we come back to this sensible position. Yet the issue keeps cropping up.

One source of this friction is that most of us niche ourselves into methodologies that we are comfortable with. And there are professional payoffs for developing methodological expertise, usually developed by doing a series of increasingly nuanced quantitative or qualitative work. What Bob's post adds to the picture (through examples, not theory!) is how knowledge creation is enhanced by combining divergent methods. Of course, people that do these combined methods are neither fish nor fowl.

Perhaps JELS or Law & Social Inquiry (current or past editors are on this thread) can advance this norm by devoting editorial space that highlights the very best empirical work that combines quantitative and qualitative work--e.g., a 5-page survey essay by the editors.

One example that Bob omitted (out of humility) was his 1988 classic, Partners with Power, which combined quantitative and qualitative work in a case study of four large Chicago law firms. I can remember asking myself whether I would ever be able to write sociolegal scholarship at this level--it provided such a rich context. Further, it has a remarkably comtemporary feel 20 years after the last interview was complete.

So I guess the takeaway is this: NLR is (or ought to be) about effective creation of knowledge--capturing law in action; methods are a means to this end, not an end in themselves.

[Stewart Macaulay]

I like Michael's comment and agree with it. Nonetheless, I'm left with a question that he or someone else might be able to answer.

As part of writing a talk that I'm to give in October, I've tried to compare the Journal of Empirical Legal Studies with the new International Journal of Law in Context. In trying to consider what the social study of law is about, these two journals take very different approaches. Obviously there is room for both. But JELS emphasizes the quantitative while IJLC largely, but not enitrely, does the literary or philosophic turn.

But I wonder to what extent, if at all, is JELS open to qualitative studies? The editor's statement in the first issue can be read several ways. It says "JELS seeks to encourage, promote, and provide an impetus for the careful collection of empirical data and the dispassionate, rigorous testing of empirical hypotheses. The central purpose of JELS is to add to knowledge of the legal system based on observation or empirical analysis, including experimental analsis. . . " We could point to "rigorous testing of empirical hypothesis," or we could jump on the term "observation." What did the author mean by "rigorous?" What was his/her idea of what is "empirical?"

Nonetheless, if we look at all of the articles in Volume 1 (Vol. 2 is being bound and I can't get it now), we find tables of quantitative data and statistics as the only approach that I see in all of the articles (perhaps Lawrence Friedman's is an exception). Perhaps we could make a case that we need a journal that encourages more hard quantitative work based on state-of-the-art statistics. Perhaps more qualitative stuff should find a home in the Law & Society Review or Law and Social Inquiry. Perhaps the editors did not have a chance to consider qualitative approaches because no one submitted such papers. But I'm left with the question whether JELS favors quantitative methods and papers about questions that can be approached by using those methods. (Interestingly, after looking at the formal rules and the text in the journal and then counting types of articles that were published, we are still left wondering what the patterns we've found mean. We need an informant).

I hope that it is not necessary to say that I like quantitative work, and I am not afraid of statistics. But I am curious about the meaning of "empirical" in the Journal of Empirical Legal Studies.

[Michael Heise]

Unlike many others, I must admit that I have not invested much time dwelling on Beth's interesting question. Here's why. First, I'm just not aware of any "ELS effort" at least as it specifically relates to "feelings about" qualitative research (or, for that matter, any other issue). Second, rather than debates about what constitutes "quantitative" or "qualitative" research, more interesting to me is the *quality* of the research and the appropriateness of the method(s) used to the research question, design, and data. Third, similar to Beth (and others), I've always been of the mind that different methodological approaches possess different blends of strengths and weaknesses and that none possess an exclusive lock on advancing knowledge. To be sure, certain research questions, designs, and data might lend themselves more appropriately to one methodology or another. But using "multiple methods," where appropriate and helpful, strikes me as a good idea. Finally, the scholarly marketplace of ideas will supply some filtering over time. Without knowing ex ante which particular methodology (if any) will survive the proverbial test of time, however, why not spread risk and learn from as many as possible?

[Elizabeth Mertz]

Bob's description fits with my experience as a linguistic anthropologist; I've found it very useful to combine different methods. Folks in my field regularly combine counting things (for example, how often particular linguistic features appear in the speech of people from different class backgrounds) with qualitative observation of the social significance of these variations. In my dissertation fieldwork, I performed a linguistic census in the two Gaelic-speaking communities in which I worked, in order to assess proficiency levels. But that would have been relatively meaningless if I hadn't also lived Cape Breton for over a year, learning to understand what different linguistic practices meant within the communities. The participant observation also aided my census work, because people were much more forthcoming after I'd been there a while. And I could better interpret small cues during the interviews so that I could encourage people to keep talking.

One question I've been wondering about is how the scholars who've been central to ELS feel about qualitative research. I'd be interested in hearing how receptive the ELS effort is to using / publishing / performing research gathered through qualitative methods (e.g., interviewing, participant observation, historical research, case studies).