Empirical Legal Studies

NYT, MTurk, and Data

Yesterday's New York Times ran a lengthy piece (here) on Amazon's Mechanical Turk ("MTurk") platform. The author joined the ranks of "Turkers" for a brief period and did some "Turking" to supplement his background research for the article. While the piece reflects a decidedly labor-economic focus, insofar as "MTurk"-created data remain a staple for an increasing slice of experimental psychological research (despite earlier concerns raised years ago by, among others, Dan Kahan (see, e.g., here and here)), the NYT''s article warrants a quick read.

Public Innumeracy

While not squarely in the typical ELS wheelhouse, the following excerpt just stopped me in my tracks.

"In her excellent book, Race to the Top, the journalist Elizabeth Green tells a story of a new hamburger that the A&W Restaurant chain introduced to the masses. Weighing 1/3 of a pound, it was meant to compete with McDonald’s quarter-pounder and was priced comparably. But the 'Third Pounder' failed miserably. Consultants were mystified until they realized many A&W customers believed that they were paying the same for less meat than they got at McDonald’s. Why? Because four is bigger than three, so wouldn’t ¼ be more than 1/3?"

To be sure, this degree of innumeracy is not typically present in law school classrooms (or I certainly hope not). That said, a general ambivalence (at best) or aversion (at worst) towards all things quantitative shapes the stream of students who self-select into law schools.

Perils of Self-Reported Data

I often caution students contemplating using self-reported data and urge them to consider observational data where possible (and appropriate to the research question and design). Why Bother Asking? The Limited Value of Self-Reported Vote Intention, by Rogers, Todd Rogers (Harvard--Kennedy School) and Masa Aida (Greenberg Quinlan Rosner), contribute empirical support to my intuition, certainly as it relates to predicted voting behavior. The findings' implications for an array of sub-fields are important. An excerpted abstract follows.

"Phone surveys from three elections, including one survey experiment, are analyzed to compare respondents’ pre-election vote intention with their actual voting behavior using administrative records (N=29,403). Unsurprisingly, many who predict that they will vote actually do not vote. More surprisingly, many who predict that they will not vote actually do vote (29% to 56%). Records of past voting behavior predicts turnout substantially better than self-prediction. Self-prediction inaccuracy is not caused by lack of cognitive salience of past voting, or by inability to recall past voting. Moreover, self-reported recall of turnout in one past election predicts future turnout just as well as self-prediction. We discuss implications for political science research, behavioral prediction, election administration policy, and public opinion."

The Polls - Torture

Both PEW and Gallup (others?) have examined public support for torture.

In a lengthy report on a variety of issues, PEW has these results:

The full discussion of these results is on page 25 of the PEW report (page 29 of the pdf file).

Gallup editor Darren Carlson discusses a November 2005 survey here (n=515). The question was, "Would you be willing -- or not willing -- to have the U.S. government torture suspected terrorists if they may know details about future terrorist attacks against the U.S.?" The results: 38% were willing; 56% were not willing; and 6% had no opinion.

Update: Here's an ABC News/Washington Post poll from May 2004 (n=1005), which found 35% of Americans thought "it's acceptable to torture people suspected of terrorism in some cases."  The poll also covered specific techniques for "getting information from prisoners," such as sexually humiliating suspects (16% support) and not allowing a suspect to sleep (66% support).

The Polls: Video Game Regulation and the First Amendment

A minor dustup occurred this week in the world of video game politics, one that implicates public thinking on First Amendment issues. In a press release, public relations firm Hill & Knowlton (H&K) reported: 

“Sixty percent of US consumers polled agree that the government should regulate the sale of violent or mature content video games while a slim majority (51%) agrees that the government should regulate mature content itself.” (emphasis added).

The Entertainment Software Association (ESA), which opposes government regulation of video games, responded to H&K’s report with some unusually harsh words. There may be some bad blood between the two organizations based on prior dealings -- the ESA’s criticisms are here [UPDATE: H&K responds here and here] -- but the ESA didn’t discuss one significant qualification to the finding of 51% support for content regulation. A comparison of the results for two of H&K’s questions suggests there is less public enthusiasm for content regulation in this survey than the reported 51%. Here are the two relevant questions:

Q11. Do you think the government should regulate violent content in video games?

Q8. If a person running for office were to take a stand on the regulation of violent or mature content in video games, to what extent would that impact your vote for him or her?*

The Polls - Presidential Favorites

The Rasmussen Reports recently asked Americans whether they have favorable or unfavorable views of every president from George Washington to The Decider. Naturally, Washington and Lincoln are the tops. Nixon and Bush are, well, not. The winner of the "not sure" category, with 74% not sure, is the guy in the picture. (Rhymes with Tohn Jyler.)

The full results are here and here. (There is clearly an error with the results for McKinley.)

UPDATE: The McKinley error has been corrected. He received a response of "very unfavorable" from 4% of respondents, not 40%.

H/T The Washington Monthly

Seat Belts, Paternalism, and New Hampshire--"Live Free and Die?" (Blumenthal guest)

The New Hampshire State Senate recently rejected a bill that would have mandated seat belt use for all adults.  According to the media, New Hampshire is the only U.S. state without some such law.  The discussion involved, unsurprisingly, issues of paternalism and "being told what to do"--squibs from earlier stories in the New York Times noted that "For many residents, buckling up and being required to buckle up are 2 different things," and one proponent of the bill noted that the debate "harkens [sic] to the libertarian 'don't tell me what to do' streak that characterizes much of [NH] politics."  The Times stories also noted that "New Hampshire has the lowest rate of seat belt use in the country, 49.6 percent," and that "77 percent of fatal crashes in the state involved occupants who were not wearing seat belts."

Paternalism, of course (whether asymmetric, libertarian, emotional, or other), is increasingly discussed in legal academia.  Much of the connection to ELS is in the context of evaluating whether particular interventions (as just one instance, Thaler and Benartzi's SMarT investment approach) in fact improve welfare on some specific outcome.

Less empirical analysis, I think, goes to the justification of such "paternalistic" intervention, and I hoped to post a few comments in that context, using the New Hampshire example as a vehicle [sorry, pun intended] for mentioning empirical research relevant to some of the usual objections to "telling someone what to do."  Existing empirical work may indicate that we need to look a little harder at such objections--and I think providing empirical grounds for such a discussion is a fruitful effort for ELS in any context.

Offhand I can imagine at least five such objections, falling roughly into three categories.  First is the classic Millian notion that people know their own preferences, and as a result are best at identifying and choosing their own preferences.  Second is the classic autonomy argument: people's freedom to choose should be valued per se ("don't tell me what to do!") ; relatedly, people's preference for the freedom to choose should be valued per se.  Third, people learn from their mistakes, and should therefore be allowed to make them--perhaps simply for the substantive benefit of learning what is correct, perhaps to "build character."  Jon Klick and Greg Mitchell get at this third category in their recent response to libertarian paternalism (among many other points, of course).

(At least two other objections, mentioned recently by Ilya Somin on the Volokh Conspiracy, involve the accountability of chosen "experts" who make paternalistic decisions and whether those experts should get the final word.  A related one, of course, involves public choice and capture.  I'll set those aside to the extent that they involve evaluation of the means of effectuating interventions once it's decided that paternalism is appropriate [except where Somin, for instance, notes that "Although the expert is more knowledgeable than I am about technical issues in his field, I am more knowledgeable than he is about my own values"--the first category identified above].)

Again, my goal is not to take sides, but rather to point out another instance of ELS's potential to advance a substantive policy discussion.  In short upcoming posts I'll try to identify some empirical research relevant to each of the five objections above.

Nw. U. L. Rev. Colloquy & Survey Methodology

Northwestern University Law Review's on-line blog, Colloquy, features a debate this week about empirical support for the belief that white Americans hold negative stereotypes associated with viewing Asian Americans as a "model minority." According to Colloquy's editors:

"Robert Chang and Rose Cuison Villazor respond to McGowan and Lindgren's prior piece, Testing the "Model Minority Myth," which argued that empirical data do not support the belief that white Americans hold negative stereotypes associated with viewing Asian Americans as a 'model minority.' Chang and Villazor argue that McGowan and Lindgren's argument is flawed because it relies heavily on data produced by asking white respondents questions about their racial attitudes in face-to-face interviews.  Chang and Villazor assert that this methodology is unlikely to yield true data about racial attitudes, as it is infected with problems of response falsification, is unable to ferret out subtler forms of racial bias, and is limited by the 'perpetrator perspective' on racial discrimination."

The Polls - Name Two Justices

I got the full report from Zogby on the recent pop culture survey. In this survey, people were asked to name only two justices, not all nine. Sixty-one percent of people could not name any justices, whereas 15% named one and 24% named two. The press release characterizes Thomas as the “most recognized justice,” and while prior surveys support this characterization (now that O’Connor is gone), the Zogby survey on its own probably supports calling him only the “most top-of-mind justice.”

Here are the percentages indicating how often each justice was named:

Thomas	20%
Scalia	11%
Roberts	9%
Ginsburg	9%
Alito	5%
Kennedy	4%
Souter	3%
Stevens	2%
Breyer	1%

Another relevant question was asking people to name the three branches of government. Forty-two percent named all three, 17% gave an incorrect “other” answer, and 41% were “not sure.” Unfortunately, it’s not clear what exactly counted as a correct answer to this question, e.g., did “President” count for the “executive” or the “courts”  for the “judiciary”? Without this information, this question is not as interesting.

n = 1213
error: +/- 2.9%

Source: Zogby International, “Pop Culture/Political Survey: Nationwide and in Ten Metropolitan Areas” (Aug. 10, 2006).

The Polls - Another Name-the-Justices Survey

Zogby released (some of) the results from a new pop culture knowledge poll yesterday. It included a couple of Supreme Court items:

When asked to name two of Snow White's Seven Dwarfs and two of the nine U.S. Supreme Court Justices, 77% of Americans polled were able to identify two dwarfs, while only 24% could name two Supreme Court Justices. Not surprisingly, Clarence Thomas, whose nomination was marked by controversy, was the most recognized Justice -- identified twice as often as his next best-known peer on the Supreme Court -- Antonin Scalia.

And the other one:

Surprisingly, despite record-breaking ratings, when asked who the newest American Idol is as of May 2006, only 23% of those polled named Taylor Hicks. Still, while that may seem low, it was double the percentage of those who recalled which Supreme Court Judge was confirmed in January 2006. Only 11% answered "Samuel Alito."

As best I can tell, Zogby has not posted the full results of this survey, but the press release is here. It contains additional results that may be interesting but that are not really ELS relevant, e.g., "Krypton was identified correctly by 60% of the respondents as Superman's planet of origin, while only 37% were able to name Mercury as the closest planet to the sun." (When the new movie came out a few months ago, I actually discussed this very question of how many people would be able to name Krypton as Superman's home planet with a friend of mine. Now, thankfully, there are data to resolve this question.)

My collection of previous name-the-justices poll results is here. I see that I still need to update it with a fourth poll from 1989; I had forgotten about it when I did the original post.

UPDATE: William Child at TortsProf points to a very nice analysis at the Language Log of how these public knowledge surveys can be spun by those who write press releases. A second Language Log post on the subject can be found here. Both posts focus on the First Amendment survey sponsored by the McCormick Tribune Freedom Museum.

The Polls - Occupational Trust

The Harris Poll has some new data on the most and least trusted occupations. Respondents were asked, "Would you generally trust each of the following types of people to tell the truth, or not?"

Doctors are at the top with 85% percent generally trusting them. Actors are at the bottom with only 26%. A few other categories:

Teachers: 83%
Professors: 75%
Judges: 70%
The President: 48%
Members of Congress: 35%
Lawyers: 27%

Additional information, including the results for all 22 occupational categories, can be found here. Unfortunately, the results do not reveal what happens when you combine a professor and a lawyer.

The Polls - Homosexuality and Gay Marriage

With the President about to make an announcement in support of a constitutional amendment to ban same-sex marriage, a just updated AEI report entitled Attitudes about Homosexuality and Gay Marriage should be of interest. Here is the abstract:

Updated for Spring 2006, this study is a compilation of public opinion polls on acceptance of homosexuality, gay marriage, civil unions, partner benefits, party identification and voting of gays, employment, and adoption. The study includes all of the latest polling data as well as important historical trends for comparative purposes.

The report is 52-pages long with numerous polling results. Assorted highlights from an earlier version of the report can be found here. The full text in pdf form of the current version can be found here.

(via Andrew Sullivan)

The Polls - NSA Surveillance

Mystery Pollster Mark Blumenthal comments on the various polls here on the CBS News PublicEye blog (rather than his own blog), and he provides links to the full results from each poll. It's a good discussion, though I question his final comparison of two poll items in the penultimate paragraph.

As a side note, view the National Security Agency Kid's Page here. Apparently, this has been around for at least a few months. Somehow, I missed it. The NSA says it's fun for adults too.

The Polls - Abortion

Gallup released an analysis today focusing on the relationship between religion and opinions on abortion.  It is not exactly full of surprises, but here is the conclusion:

[K]nowing an individual's frequency of church attendance helps predict his or her attitude toward abortion, above and beyond what can be predicted by knowing that individual's partisan affiliation. Republicans who attend church regularly are the most likely to be anti-abortion, while Democrats who seldom or never attend church are the least likely to be anti-abortion.  Both variables are important in understanding the nature of abortion attitudes in America today.

The full article is available here.

The Polls - Modern Morals

Last week, the Pew Research Center released a report last week entitled, “A Barometer of Modern Morals: Sex, Drugs, and the 1040.”  Respondents were asked to classify ten behaviors as “morally acceptable, morally wrong, or not a moral issue.”  The following results provide the percentages of people classifying each behavior as morally wrong:

Married people having an affair	88%
Not reporting all income on taxes	79%
Drinking alcohol excessively	61%
Having an abortion	52%
Smoking marijuana	50%
Homosexual behavior	50%
Lying to spare someone's feelings	43%
Sex between unmarried adults	35%
Gambling	35%
Overeating	32%
N = 745
Sampling error: +/- 4%
Interview dates: February 1-5, 2006

The complete report, with breakdowns by age, gender, education, ideology, etc., is available here.  A one-page overview that includes breakdowns only by education levels is available here.  Note that this overview lists a misleading n of 1502, which is the number of respondents for a larger survey.  Just short of one-half of these 1502 respondents were asked the “modern morals” questions on which this report is based.