Empirical Legal Studies

Testing a "Ratio Problem" in Finance Scholarship

On the heels of a recent post (here) describing a Stata regression command tailored to the unique needs of a dependent variable that takes the form of a ratio/percentage/fraction, comes a recent paper that explores potential problems with finance papers that model ratios. Potentially compounding the problem is that ratios as dependents variables are certainly not uncommon in the finance literature (see, e.g., papers featuring Tobin's Q). In The Ratio Problem, Robert Bartlett (Berkeley) and Frank Partnoy (Berkeley) frame the ratio "problem" in terms of two challenges–omitted variable and measurement error bias–"that arise anytime a researcher uses linear regression to estimate a production function that has a ratio as an output." As the paper notes, "In theory, the [ratio] denominator is not necessarily problematic; in practice, however, there are statistical concerns whenever the output of a production function is a ratio." The paper's abstract follows.

“We use the term ‘ratio problem’ to describe the omitted variable and measurement error bias that can arise when a ratio is the dependent variable in an economic model. First, we show how bias can arise from the omission of two classes of variables based on a ratio’s denominator. As an example, we demonstrate that the widely-cited ‘inverse U’ relationship between managerial ownership and Tobin’s Q is reversed when these variables are included. Second, we show how measurement error in the ratio’s denominator can produce bias. We provide empirical tests and solutions, and urge caution about ratios as dependent variables.”

Modeling Rates/Fractions/Percentages

While I probably should have mentioned this when Stata's (relatively) new and quite helpful "fracreg" command was released, a recent surge of student questions about how to best model a rate (or fractional) dependent variable motivates this post.

Periodically ELS scholars will need (or want) to model a defendant variable that is bounded between zero and one (e.g., rates, fractions, percentages, etc.). Until recently few satisfactory options (at least "off-the-shelf" ones) existed as the various standard "reg" commands are inapt given the rate/fractional/percentage dependent variable's bounded structure. The "fracreg" command in Stata, however, is designed for dependent variables that range from zero to one (inclusive). Anyone intrigued should consult the user manual (here) and watch a brief introductory tutorial video (here).

Comparing "predicted probabilities" and "odds ratios"

Those who use the logistic as well as the margin commands (and both are quite common in ELS) sometimes confront the question of whether to report out odds ratios (from the logistic command) or the predicted probabilities (from the margin command). As one commentator notes in this Stata list discussion (here), "An odds [ratio] is related but not the same as a probability. They are related in the sense that both are ways of quantifying how likely an event (success) is. They differ in that the "odds [ratio] is the average number of success per failure. The probability is the average number of successes per trial." What is the more intuitive of the two will, of course, vary across people.

All About Multilevel Modelling Resources

With an admitted bias for "one-stop" resource sites, I recently came across a very thorough (and thoroughly helpful) site sponsored by the University of Bristol's (UK) Centre for Multilevel Modeling (here). Multilevel modelling's usefulness for many ELS research projects is as acute as it is obvious. The site teems with an array of focused resources, including a deep pool of useful online instructional videos (here). For anyone interested in a brief overview of multilevel modelling, this quick explanatory video (here) might be one place to start; a series of PowerPoint presentations (found here) is another.

Interrupted Time Series Regression

I note this recent discussion in Statalist (click here) only because an interrupted time series research design is not infrequently helpful in empirical legal studies. Note also comment No.7's point about the assumption that the external shock (the "interruption") applying equally across all the firms in the panel.

Assessing Normal Distributions

A nice illustrative discussion (here) on various approaches to assess whether "normal" distribution assumptions plausibly hold. Participants in the discussion provide a helpful illustration (below) on two perspectives (kernel density vs. qnorm) on a possible bi-modal distribution.

Law School Class Recordings and Student Access Policies: Data From Down Under

Recording classes and granting students liberal access to them are increasingly the norm in universities and law schools around the world. At the same time, however, mounting anecdotal evidence implies that increasing student access to class recordings corresponds with decreasing student class attendance. A recent paper, If You Record, They Will Not Come--But Does It Really Matter? Student Attendance and Lecture Recording at an Australian Law School, by Natalie Skead (Univ. Western Aus.) et al., presents and discusses results from a study that set out to test this “conventional wisdom” with a bit more rigor.

The paper is noted for its mixed methods which include a student survey, focus groups, and a manual count of observable student class attendance. Not only do the researchers' data speak to their specific research question (that is, the relation between student attendance rates and class recording access) but they also permit researchers to assess a derivative question—whether discrepancies exist between student attendance self-reports and actual observable student class attendance.

What the researchers found is that student access to class recordings is “strongly associated” with decreased student class attendance. As it relates to the reliability of self-reported survey data, the paper notes that the results “indicate a disparity between the high rates of self-reported attendance and the actual attendance count at lectures.”

Three quick closing points. First, while unlikely germane to the findings, it remains important to note that in Australia, and elsewhere (but unlike in the United States), law school includes both undergraduate as well as graduate students. Second, even assuming, arguendo, that the paper’s core findings hold, specific policy consequences relating to student access to class recordings are not obvious. Third, this research illustrates, yet again, why researchers need to approach self-reported survey data with caution given persistent reliability concerns.

Simple Barcharts Options in Stata's Complex Graphics Coding

While I obviously should have stumbled onto this earlier, I'll continue to ascribe it to Stata's unnecessarily (in my view) complicated graphics coding. Indeed, complications hamstring even the "simple stuff."

The example below involves "style" elements incident to the standard, simple barchart. The conventional "graph hbar" command generates the "typical" barchart (on the left). The "statplot" command (user-written, requires downloading), however, generates (to my eyes, anyway) the slightly less "cluttered" and more "intuitive" barchart on the right. (Click here for additional information on "statplot".)

To be sure, this admittedly subtle distinction pivots on mere aesthetics and, of course, neither barchart chart is "wrong" in any way. They simply convey needed information in slightly different ways. But if I took anything from Tufte's classic exposition, The Visual Display of Quantitative Information, at some point "mere aesthetics" can matter. Or, if not "matter," can at the very least help or improve.

Legal Impact Research & Evidence of Racial Profiling

Far too many legal and court scholars (as well as law students, lawyers, judges, etc.) simply assume, at least as a presumptive matter, that court decisions yield the outcomes they seek. Not enough recognize how institutional arrangements, incentives, the "law of unintended consequences," etc., inform how court decisions are implemented "on the ground." A recent paper, An Empirical Assessment of Pretextual Stops and Racial Profiling, by Stephen Rushin (Loyola-Chicago) and Griffin Edwards (UAB-Economics), provides yet another helpful and timely reminder of the efficacy of subjecting formal legal doctrines to data and rigorous research designs.

In their paper the authors exploit a change in Washington State law concerning police traffic stops. While a 1999 Washington State Supreme Court decision (Ladson) barred pretextual traffic stops in 2012 the Washington Court reversed course (Arreola) by permitting "mixed motive" traffic stops. Varying this legal rule opened a door for a DID research design that draws on 8.2M traffic stops in Washington State between 2008 and 2015. And what the results suggest is that the Arreola court decision "was associated with a statistically significant increase in traffic stops and searches of nonwhite drivers relative to white drivers. We find that most of the increase in traffic stop of non-white drivers after Arreola occurred during the daytime, when police officers could more easily ascertain a driver’s race." Overall, this paper will interest not only because of the massive dataset but also for its empirical specification.

Standard (OLS) Regression for a *Binary* Dependant Variable?

Evidently, the question is not as crazy as I likely would have assumed 40, 30, or, perhaps, even 10 years ago. In a widely-discussed draft paper (click here), Logistic or linear? Estimating causal effects of treatments on binary outcomes using regression analysis, Robin Gomila (Princeton--(grad student) social psych) makes a case for OLS. While Andrew Gelman (Columbia--statistics) is among those who disagree (here), he also recommends alternative parallel suggestions and approaches. To the extent that the Gomila paper is partly motivated by the sometimes "muddled" interpretation of coefficients after a logistic transformation, my quick sense is that altering the empirical strategy risks a potential over-reaction.

New Goodness of Fit ("GOF") Test

Testing whether two separate distributions systematically vary typically involves a t-test or ksmirnov test. These two popular GOF tests involve comparing two distributions' means (t-test) or a hypothesized equal distribution (ksmirnov). A recently-released user-written command in Stata, distcomp, moves the needle a bit for GOF testing by assessing the equality of the distribution functions point-by-point. That is, instead of a single rejection (or non-rejection), the distcomp test generates ranges of values in which the two distributions' difference may achieve statistical significance.

To locate and download distcomp use the "findit" or "ssc install" functions in Stata. For a much more thorough explanation and description of the distcomp command click here.

Variable "Skew"

Any discussion about what to do with a "highly-skewed" variable needs to immediately differentiate between "highly-skewed" dependent and independent variables. A "highly-skewed" (or "non-normally distributed") dependent variable (or, more accurately, as it relates to the errors) raises all the obvious questions when it comes to the appropriateness of an OLS regression specification given its underlying core assumptions. A highly-skewed independent variable, by contrast, raises none of these particular concerns. As one of the commentators notes (see discussion, and an illustration, here), in certain contexts a highly-skewed independent variable may simply reflect "just the way the data are."

"Inflated" Standard Errors

While what constitutes an "inflated" standard error remains contested, what is clear, however, is that there is no standard, "go-to," statistical "fix" for any "inflated" standard errors. As one commentator notes (see StataList discussion here), "they are what they are." Rather, a more helpful approach towards "inflated" standard errors is to assess what in the underlying data set (or model) might be driving the standard errors. The standard set of "usual suspects" includes, e.g., outliers in the data set, an "extreme" level of multicollinearity, or an "over-fit" model. (For more on "over-fitting" regression models, see here.)

Outliers

While there are no obvious "hard-line" rules when it comes to identifying outliers in a data set--let alone what to do with any that are identified--a recent wide-ranging discussion is found here. Moreover, Professor Williams (Notre Dame--sociology) has put together an extremely helpful and rich resource document (here). Finally, another possibly useful listserv discussion (albeit a dated one) is found here.

NYT, MTurk, and Data

Yesterday's New York Times ran a lengthy piece (here) on Amazon's Mechanical Turk ("MTurk") platform. The author joined the ranks of "Turkers" for a brief period and did some "Turking" to supplement his background research for the article. While the piece reflects a decidedly labor-economic focus, insofar as "MTurk"-created data remain a staple for an increasing slice of experimental psychological research (despite earlier concerns raised years ago by, among others, Dan Kahan (see, e.g., here and here)), the NYT''s article warrants a quick read.

"Zeros" Problems

While not unique to empirical legal scholarship, how to deal with a "zeros problem" is not uncommon, especially given the nature of many research designs and questions in legal scholarship. To take one obvious--and well-trod--example, efforts to model punitive damage awards need to account for their absence in the over-whelming majority of civil cases decided by courts. A recent discussion on the StataList (here) provides yet another take on the array of challenges posed by "zeros" in data sets. The resultant discussions emphasizes, in part, that researchers need to assess with accuracy and clarity, ex ante, precisely what a "zero" means in the context of their data sets. As well, for a recent take on differences between Heckman selection and two-part models, a 2015 JELS paper (here), by Eisenberg et al., remains current.

"Paired" v. "Unpaired" T-Test

Efforts to compare the means from two sample populations with a standard t-test invariably bump into a "paired" versus and "unpaired" decision. Whether a "paired" or "unpaired" comparison is appropriate pivots on a subtle distinction that can be difficult to describe. A recent Statalist discussion might provide a helpful way to think about this distinction.

The discussion (here) began with the following (edited) question: "If there is economic growth data about two countries over the same thirty years, should it be considered as paired or unpaired data for two sample t-test?"

The various responses included the following: "For unpaired tests imagine 30 men and 30 women completely unrelated and compare their heights. You are comparing the mean male height and mean female height and the context is the variability of male heights and of female heights. For paired tests imagine 30 heterosexual married couples and compare height for each husband with each wife."

With this in mind, the answer to the question posed is: "Your panel example is still paired because there is pairing by years."

Log Transformations & the "Zeros Problem"

Strategies to address normal distribution assumption concerns can quickly lead to the log transformation of variables. However, what to do with "zeros" in a variable requiring log transformation persists as an under-studied issue. And one well-known example, punitive damage awards, makes clear that such "zeros problems" are not uncommon in many empirical legal research projects.

In Dealing With the Log of Zero in Regression Models, the authors, Christophe Bellego (Center for Research in Economics and Statistics) and Louis-Danile Pape (Center for Research in Economics and Statistics), engage with this nettlesome issue head-on. The brief (15 pp.) paper's abstract follows.

"Log-linear and log-log regressions are one of the most used statistical model. However, handling zeros in the dependent and independent variable has remained obscure despite the prevalence of the situation. In this paper, we discuss how to deal with this issue. We show that using Pseudo-Poisson Maximum Likelihood (PPML) is a good practice compared to other approximate solutions. We then introduce a new complementary solution to deal with zeros consisting in adding a positive value specific to each observation that avoids some numerical issues faced by the former."

Instrumental Variables Regression--A Helpful Intro

For a brief (5+ min) introduction to--or a very quick refresher on--the basics of instrumental variables regression, including some post-estimation commands, one could do far worse than this Stata video. The video walks viewers through the basics on how to fit an instrumental variables model for endogenous covariates (using the "ivregress" command).

"p-Hacking" explained

One consequence of the on-going "replication" crisis in the social (and, increasingly, hard) sciences is a spike in the salience of the term "p-Hacking." The term "p-Hacking," as defined by Robert MacCoun (Stanford) in his recent essay, p-Hacking: A Strategic Analysis, “joins a growing taxonomy for describing the myriad ways in which ‘cold cognitive’ and ‘hot motivational’ biases can creep into the interpretation and use of research findings.” In his brief essay MacCoun dives deeply into the issue and proposes “four different models of p-hacking behavior and examining what kinds of aggregate patterns they might be expected to produce.” The abstract follows.

"The phenomenon of p-hacking occurs when researchers engage in questionable practices that enable them to report findings as being statistically significant. I offer four models of p-hacking behavior – unconditional, strategic, greedy, and restrained – and explore the implications of each model. I then discuss the implications of recent reforms (routine replication, pre-registration, and blinded data analysis) with respect to these models."

MTurk and Evidence on Measurement Error

While questions surrounding the potential for systematic measurement error from MTurk samples cannot be easily resolved in a single paper, given the popularity of MTurk samples in many fields, notably psychology, careful attention scholarly is warranted. The initial "warning" about MTurk samples came in 2013 from Dan Kahan (Yale), particularly for studies seeking to test “hypotheses about cognition and political conflict over societal risks and other policy-relevant facts.” Since then we've seen a developing literature that sets out to test potential limitations to MTurk samples. A recent paper by Douglas Ahler (FSU-Poli Sci ) et al., The Micro-Task Market for Lemons: Data Quality on Amazon’s Mechanical Turk, focuses on the potential for measurement error, especially Type II errors, injected by, among other factors, the "presence of 'non-respondents' (bots) or non-serious respondents on the platform." The paper's core findings include:

"While we find no evidence of a 'bot epidemic,' we do find that a significant portion of survey respondents engaged in suspicious behavior. About 20% of respondents either circumvented location requirements or took the survey multiple times. In addition, at least 5-7% of participants likely engaged in 'trolling' or satisficing. Altogether, we find about a quarter of data collected on MTurk is potentially untrustworthy. Expectedly, we find response quality impacts experimental treatments. On average, low quality responses attenuate treatment effects by approximately 9%. We conclude by providing recommendations for collecting data on MTurk."

When--and How--To Use Kruskal-Wallis

Helpful discussion, and example, of how to execute the Kruskal-Wallis ("K-W") test as well as an explanation on why one would turn to it (click here). In general (and in brief), folks typically turn to the K-W test when you have one nominal variable (e.g., gender) and one measurement variable (e.g., test scores). While one-way anova is used when the measurement variable distributes normally, K-W is used when the measurement variable does not.

Balanced v. Unbalanced Panel Data; "reg" v. "xtreg" Commands--Asking the Right Question

An admittedly small but nonetheless important point emerged in a recent StataList discussion (here). While the original query wondered whether a decision between the "reg" and "xtreg" commands pivoted on whether panel data were balanced or unbalanced, a very helpful commenter quickly made clear the question's pivotal (and mistaken) assumption. To be sure, while the "reg" and "xtreg" commands may yield the same or similar results in certain conditions, they estimate different models and different parameters. In general, where panel data are involved (balanced or unbalanced), as the commenter notes,"-reg- is simply inappropriate."

"georoute"--Measuring Distance and Time Between Two Points

While requiring a user-download, a new--albeit quirky--command, "georoute", allows researchers to compute "travel distance and travel time between two points defined by their addresses or their geographical coordinates." Or, more specifically:

"calculates the georouting distance between two addresses or two geographical points identified by their coordinates. It uses the HERE API (https://developer.here.com) to retrieve distances in two steps. In the first step, addresses are geocoded and their geographical coordinates (latitude and longitude) are obtained. In the second step, the georouting distance between the two points is obtained. The user can also provide directly geographical coordinates, which will bypass the first step."

Such a program may prove useful, e.g., to researchers exploring various "access to justice" research papers that seek to exploit/explore data on physical distance. A brief discussion is found here.

For those interested, to download this program, from within Stata, type: ssc install georoute

Model Options For Log-Transformed Dependent Variables

This month's Stata News (34:2) includes a quite helpful (and instructional--with examples and coding) discussion about some classic challenges that arise from a log-transformed dependent variable. First, the need to log-transform dependent variables is exceedingly common in ELS (e.g., punitive damages models). Second, when log transformations are necessary interpretation challenges arise as few among us think in terms of, e.g., "log punitive damages" or "log wages." The essay walks readers through various model options (regression and poisson) and how to exploit the margins command to assist with interpreting results even further.

How to Deal with an Ordinal Dep Var When You Need/Want to Include Fixed Effects?

A recent question posed on StataList has stimulated an interesting discussion (here) as well as a call for FE ordinal models. The specific question:

"So to sum up my question: Is it possible to 'argue' that my ordinal dependent variables can be treated as continuous and I can therefore use FE for them as well? If yes - could you point me towards the appropriate literature/methodology? Or is this not possible and I need to use logit/probit for the ordinal dependent variables?"

The core of the question--can an ordinal dependent variable be treated as "continuous"--comes up most frequently in the survey context where a response involves a Likert Scale (typically a 5- or 7-point scale -- or an even-numbered scale that pushes respondents off a mid-point response). While reasonable minds can--and do--differ on what modeling options may plausibly exist, at the very least, and as a first step, a researcher would need to closely examine how any ordinal dependent variable distributes as this may itself limit model selection options.

"Significance" in Marginal Effects v. "Original" Model

A lively discussion (here) on the Stata list takes up the question: "This variable has significant effects in the original (logit/probit/heckprobit/whatever) model. But its marginal effect is not significant. Why?"

As the ensuing (and on-going) discussion makes clear, this nuanced discussion involves, among other things, careful attention to precisely what specific hypotheses are being tested in a margins specification versus an "original" specification.

Data on Post-Conviction Criminal Expungement

As briefly described in a recent NYT op-ed (on March 20, 2019; link here for those who can get behind the NYT pay-wall), Professors J.J. Prescott (Mich.) and Sonja Starr (Mich.) submit an increasingly discussed policy--a push for increased expungements of criminal convictions--to data. As the paper notes, motivating the increased policy attention to expungements are concerns about "reentry barriers faced by people with criminal records." Contributing to an under-developed empirical literature on the possible impacts of criminal expungements is a paucity of public information on expunged criminal records. In Expungement of Criminal Convictions: An Empirical Study, Prescott and Starr lever their access to "complete, deidentified criminal records from the Michigan State Police on all individuals who had obtained criminal record “set-asides” (Michigan’s term for record-sealing) as of March 2014, as well as full criminal history records for much larger comparison groups of individuals with convictions that were not set aside."

The paper's careful attention to research design addresses one (but not all) traditional threat to identification posed by selection bias. By adopting a within-person design--and not comparisons between those received expungements and those who did not--the specification's identification is not distorted by the likely (almost assured) systematic differences between expungee and non-expungee sub-groups. That said, even the paper's within group design can not fully blunt potential distortions owing to variations in the timing of an individuals' expungement application. A summary of key findings is included in the abstract.

"Laws permitting the expungement of criminal convictions are a key component of modern criminal justice reform efforts and have been the subject of a recent upsurge of legislative activity. This debate has been almost entirely devoid of evidence about the laws’ effects, in part because the necessary data (such as sealed records themselves) have been unavailable. We were able to obtain access to deidentified data that overcomes that problem, and we use it to carry out a comprehensive statewide study of expungement recipients and comparable non-recipients. We offer three key sets of empirical findings. First, among those legally eligible for expungement, just 6.5% obtain it within five years of eligibility. Drawing on patterns in our data as well as interviews with expungement lawyers, we point to reasons for this serious “uptake gap.” Second, those who do obtain expungement have extremely low subsequent crime rates, comparing favorably to the general population—a finding that defuses a common public-safety objection to expungement laws. Third, those who obtain expungement experience a sharp upturn in their wage and employment trajectories; on average, within two years, wages go up by 25% versus the pre-expungement trajectory, an effect mostly driven by unemployed people finding jobs and very minimally employed people finding steadier or higher-paying work."

Longitudinal Study of Lawyer/Life Satisfaction: UVa Law Class of 1990 Over Time

Building on an earlier survey (conducted in 2007) of Univ. of Virginia Law School students (class of 1990) 20 years out, in Lawyers at the Peak of Their Careers: A 30-Year Longitudinal Study of Job and Life Satisfaction, forthcoming in JELS (Vol.16:1), John Monahan (UVa) and Jeffrey Swanson (Duke) re-surveyed (conducted in 2017) those responding individuals 30 years out to gain some sense of how professional and life satisfaction may have evolved over time. Interacting the 2007 and 2017 response rates suggests that 58 percent of the (still living) UVa Law class of 1990 participated in both survey waves. While questions about how much one may reasonably generalize from a self-selecting sample of a single graduating class from a single elite law school certainly persist, that such longitudinal survey work is unusual evidences the degree of difficulty incident to such research endeavors. A summary of key findings follow from an excerpted abstract.

"We found respondents to have taken diverse career paths, with no single work setting accounting for more than one‐quarter of the respondents and with fully one‐third of the respondents changing jobs in the past decade. Marked gender differences in the professional lives of respondents persisted (e.g., women continued to be much more likely than men to forego full‐time employment 'in order to care for children' (30 percent vs. 4 percent)). Working conditions at large private law firms stayed problematic, with the portion of respondents negatively affected by a stronger stress on economic sustainability being twice as high among those working in large firms (77 percent) than among those working in other settings (38 percent). Finally, both career satisfaction and life satisfaction again were found to be high, with 77 percent of respondents satisfied with the decision to become a lawyer, and 91 percent satisfied with their lives more broadly."

"heckman" v. "etregress"

Two-stage modeling if not uncommon in ELS and relevant Stata coding has become quite specialized for subtly distinct contexts. A recent discussion on the Statalist (click here) provides a helpful (if brief) clarity.

The question posed sought modeling advice on estimating "the effect of having a female director on board on earnings management. I have a continuous variable for earnings management as the dependent variable. I want to first predict the probability of having a female board member versus having an all-male board. I am doing this to mitigate the sample selection bias."

The discussion elicits a distinction between the proper use of the "heckman" and "etregress" two-stage commands. While the "heckman" command is used when you observe outcomes for only the treated variable of interest, " the "etregress" command, by contrast, is used when "some outcome is observed for all units."

Power and More Power (Testing)

Power and sample-size calculations are an important part of planning a scientific study. In addition, too often papers that report (perhaps unexpected) null results do not take the next step and assess the possibility that any null results may be ascribed to an underpowered sample. While simple power tests are understood by many (click here for an inventory; click here for a helpful general intro video), power testing for more complex empirical specifications are not. A recent entry (the first of a series) on the Stata Blog (here) walks readers through power testing for more sophisticated modeling.

Marginal Effects Primer

Prof. Richard Williams (Notre Dame) put together an exceedingly helpful "primer" on Stata's "margins" command. Prof. WIlliam's PowerPoint slide deck (here) walks readers through the basics, provides helpful explanations, and illustrates though an array of concrete examples. As marginal effects gain in popularity many might find this helpful.

UPDATE: A loyal Els Blog reader, Prof. Jeff Yates (Binghamton), alerted me to two addition helpful links germane to margins (here and here).

Manipulating Date/Time Variables

An often needed--yet difficult to create--variable is a "duration" variable that seeks to capture time (or number of days) and is derived by "subtracting," e.g., a start-date from an end-date. This computation's degree of difficulty is enhanced by complexities incident to how dates (and time) are formatted in Stata. Those interested can find a recent step-by-step explanation (and illustration) here.

Longitudinal Studies

When appropriate to a research question(s) and where resources permit, longitudinal studies can provide important insights that are not possible from studies that use the more common cross-sectional approach. In longitudinal studies, researchers engage the same participants with repeated measures over time. While perhaps somewhat unusual in the legal research literatures, longitudinal studies are not unknown.

A recent paper by John Monahan (UVa) and Jeff Swanson (Duke--psych), Lawyers at the Peak of Their Careers: A 30-Year Longitudinal Study of Job and Life Satisfaction, provides one such example. The study follows a cohort of UVa law students who matriculated in 1987. Critical to the success of longitudinal studies, of course, involves censoring attributable to sample participation decay over time. In their original paper the researchers (in 2007) engaged with 58% of those law students who entered UVa law school 20 years earlier in 1987. Their recent follow-up paper, reporting results from their 2017 follow-up survey, includes 81% of those who participated one decade earlier (or, to be more precise, the 2017 sample includes 81% of the 58% who participated ten years prior in 2007). A summary of key findings, drawn from the abstract, follows.

“We found respondents to have taken diverse career paths, with no single work setting accounting for more than one-quarter of the respondents and with fully one-third of the respondents changing jobs in the past decade. Marked gender differences in the professional lives of respondents persisted (e.g., women continued to be much more likely than men to forego full-time employment 'in order to care for children' (30 percent vs. 4 percent)). Working conditions at large private law firms stayed problematic, with the portion of respondents negatively affected by a stronger stress on economic sustainability being twice as high among those working in large firms (77 percent) than among those working in other settings (38 percent). Finally, both career satisfaction and life satisfaction again were found to be high, with 77 percent of respondents satisfied with their decision to become a lawyer, and 91 percent satisfied with their lives more broadly.”

Research Design and Multi-Level Modeling--An Illustration

A recent post on Andrew Gelman's blog (here)--ironically prompted by a grad student's frustration with efforts to detect treatment effects in, of all things, the weight-training context--caught my eye not for any insights into weight-training but, rather, for some of the post's practical research design and multi-level modeling suggestions.

An truncated version of the student's question follows: "Consider the field of exercise science, and in particular studies on people who lift weights... Because muscle gain, fat loss, and (to a lesser extent) strength gain happen very slowly, and these studies usually last a few weeks to a few months at most, the effect sizes are all quite small. This is especially the case when comparing any two interventions that are not wholly idiotic for a difference in means."

Gelman's multi-step reply begins with the assumption that "the way to go is to have detailed within-person trajectories. Crude between-person designs just won’t cut it...." Executing a within-person trajectory approach from a research design and data-gathering perspectives involves: "At the design stage, it would be best to try multiple treatments per person. And you’ll want multiple, precise measurements. This last point sounds kinda obvious, but we don’t always see it because researchers have been able to achieve apparent success through statistical significance with any data at all." Gelman's final suggestion focuses on model selection. "The next question is how to analyze the data. You’ll want to fit a multilevel model with a different trajectory for each person: most simply, a varying-intercept, varying-slope model."

While the example in the post features R code, it nicely illustrates some of the critical ex ante decisions necessary for successful empirical studies.

"Checking the Basics": Regression Assumptions

Too often researchers neglect the basics that they learned in grad school (or, perhaps, as undergrads) regarding regression assumptions. A recent Statalist discussion (here) highlights some of the basics when it comes to standard OLS regression "diagnostics." The commentary also underscores that there is no single "correct" approach and that level-headed judgment is at a premium. Finally, as an early comment notes, searching on "regress postestimation" and "regress postestimation plots" in Stata's "Help" tab uncovers a treasure-trove of helpful commands.

Online Opt-In Surveys, Sample Weights, and Accuracy

For some obvious and less-than-obvious reasons, polling in general and online opt-in polling in particular have come under increased scrutiny. Despite their increased popularity (owing partly to cost factors), a recent Pew Research Center Report notes that such surveys have "raised some concern within the industry because, while low participation rates pose a challenge for all surveys, the online opt-in variety face additional hurdles." In many instances researchers turn to various sample weighting adjustments in an effort to off-set sample biases. While the brief Report can speak for itself, one clear take-away is that: "Even the most effective adjustment procedures were unable to remove most of the bias." Interestingly, "when it comes to accuracy, choosing the right variables for weighting is more important than choosing the right statistical [adjustment or 'sample weighting'] method." Andrew Gelman's (Columbia--Statistics) comments on the report are found here.

Substantive v. Statistical Factors in Model Selection

In a recent Stata list discussion, a grad student investigating firms' leverage ratios raised a question about whether to run a fixed-effects or OLS model on some panel data and cluster. To assess his results' robustness to model specifications, he ran the data across an array of models (below).

The advice offered, however, emphasized that "the answer to your ... question comes from substantive finance considerations, not statistics or Stata." That is, in some instances the precise nature of your research question and context, in combination with research design and data structure factors, inform specification selection where various options are, from a purely statistical perspective, "not necessarily wrong."

D-I-D Slides

Compliments of Clyde Schechter (a prominent blogger on the Stata List) those interested in a very helpful set of basic difference-in-difference slides can click here. The slides summarize various D-I-D basic designs, key assumptions, threats to validity, and related matters.

Data Weights: To Apply, or Not To Apply?

While much depends on context and a researcher's specific objective(s), questions about whether to apply analytic weights (when such weights are made available in a data set) too often resist easy, clean answers. A discussion of this issue is ongoing on the StataList forum (with illustrative examples).

"Radar" or "Spider" Graphs

A recent thread on the Stata listserv alerted me to a user-created command, radar, that generates "spider" graphs (see, e.g., below). The radar command "produces a radar plot from at least two variables. The first variable must always contain the label for the axes and the second variable must be numeric."

While I'm not entirely sure how such graphs can be deployed effectively, I was delighted to learn that this graphic potential existed. For those interested, just use the "findit" command to locate, download, and install the "radar" graphics module.

Calculating Damages in Successful Discrimination Lawsuits

An interesting paper seeks to improve accuracy by approaching discrimination detection and discrimination remedies with distinct empirical strategies. In Improved Statistical Methods for the Calculation of Damages in Discrimination Lawsuits, Scott Susin (HUD) and Ioan Voicu (OCC) note that: "While regression techniques for detecting discrimination have generated a vast literature, similar techniques for remedying discrimination have been comparatively neglected." The authors propose Bayes techniques for compensation calculations. The paper's abstract follows.

"In this paper, we develop a new method that uses empirical Bayes techniques to more accurately incorporate the individual-specific information. While some non-injurious factors will always be unobserved, it is possible to estimate the percentage of the variance due to these factors. Given this estimate, our method calculates individual-specific damages by using empirical Bayes methods that combine a general estimate, based on a regression coefficient, with individual-specific estimates based on regression residuals. It thus combines many of the best features of previous approaches and results in considerably more accurate payments than existing statistical procedures."

Stata's Instructional Video Library

What began as a "good idea" and complemented other resources has quickly evolved into an invaluable stand-alone resource for novice and experienced Stata users alike. The current collection in Stata's instructional video library (hosted on YouTube) is now both wide and deep; I recommend this resource to all.

Plotting Marginal Effects From Logit

As the "logit" command is quite common in ELS and as many political scientists have embraced marginal effects, I thought this recent and helpful exchange (here), featuring concrete coding suggestions, might interest some readers.

Robustness Checks (and Replication)

Interesting discussion (here) about various motivations for and objectives of robustness checks. Lurking between the lines of the discussion is an effort to distinguish "standard" robustness checks from "replication" efforts. As Gelman (Columbia-Statistics) notes: "For example, maybe you have discrete data with many categories, you fit using a continuous regression model which makes your analysis easier to perform, more flexible, and also easier to understand and explain—and then it makes sense to do a robustness check, re-fitting using ordered logit, just to check that nothing changes much." From my way of thinking, robustness checks evidence (to editors or readers) that a paper's main substantive findings are not solely "model- or assumption-specific."

Gelman and Carlin's 'Type M' and 'Type S' Errors and Stata's rdesigni

Increasingly "fed up" the concepts of "'power,' 'type 1 error,' and 'type 2 error,' because all these are defined in terms of statistical significance," Andrew Gelman (Columbia--Statistics) developed and advanced "Type S (Sign) and Type M (Magnitude) Errors." Gelman, conceding that "concepts of Type S and Type M errors are not perfect," nonetheless concludes they represent a helpful "step forward." Gelman (with John Carlin) described his thinking in a 2014 paper, Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. As well, the paper is summarized in a more recent blog post (here).

While few are ready to completely jettison more traditional power testing, Gelman and Carlin's alternative "Type S and Type M" struck many as a useful complement. Additional evidence of the possible promise of "Type S" and "Type M" tests is found in the recent release of Daniel Klein's user-written Stata command add-on, entitled rdesigni. (For some background on the new rdesigni command click here for a recent list-serv discussion.)

Those already equipped with Stata and wanting to explore this new command need only execute to following commend (with Stata launched):

ssc install rdesigni

C.I.s, Overalpp, and Statistical Significance--A Quick Reminder

An asymmetric relation between CIs (confidence intervals) and statistical significance constantly trips up students (and others). A quick-and-dirty summary follows (a fuller discussion, with helpful examples, is found on the Stata Blog here).

In short: if the individual CIs do not overlap, then the difference is statistically significant. However, the reverse is not necessarily true. That is, CIs for individual parameters can overlap even if their difference is statistically significant. What is also true is that if the difference is statistically significant, the CI for the difference will exclude zero.

How To Generate 'Marginal Effects' Estimates For Sub-Groups in Your Data

In a scenario described in a recent exchange on the Stata List, a researcher seeks to accomplish a discrete task: specifically, to "estimate some marginal effects for different income levels" (and in the coding the "group" var identifies the various income groups ["group==1", group==2", etc.]). The researcher correctly identify three separate--though similar--coding approaches toward the desired task. The three separate coding approaches yield slightly distinct sets of results (in the scenario, two). Consequently, as a commentator notes, the correct coding approach for any given task "depend[s] on your research goals, [and] you have to choose the one that fulfills them." One important take-away is that is often critical to understand how subtly different coding approaches--each pursuing a common objective--can yield results that differ in critical, yet subtle, ways. As a commentator observes: "We are all guilty, much of the time, of throwing around the term 'marginal effects' without specifying which of the infinitely many different marginal effects are associated with any model" (emphasis added).

F-test for Difference Between Two Coefficients in a Regression Estimation

To quickly summarize a discussion on the Stata Blog (here), for those seeking to test whether coefficients for two independent variables in the same regression estimation systematically differ from one another, the basic coding structure follows:

reg a b c

test b = c

A resulting F-test statistic that falls below your significance threshold implies that the difference between the coefficients for independent variables b and c is significant. Obviously, a suite of other post-estimation commands also exists for a host of other issues.

Non-Parametric Regression

While I haven't yet had a chance to use it myself, the recently-released Stata v.15 includes the npregress command. Nonparametric regression, similar to the more traditional parametric regression, predicts a mean of an outcome for a given set of covariates. What separates parametric and nonparametric models, however, includes assumptions about the functional form of the mean conditional on the covariates. In particular, unlike parametric models, "nonparametric regression makes no assumptions about the functional form." Thus, output from nonparametric models supports valid inferences "regardless of the true functional form."

A recent Stata Blog post (here) describes the new command in much greater detail and provides helpful comparisons illustrating output differences between parametric and nonparametric models.