In this two-part post, I want to discuss two types of interactions I have witnessed many ELS scholars experience -- the interaction between empirical legal scholars and scholars from other relevant disciplines, and the interaction between empirical legal scholars and legal scholars. In the first part of the post, I will mostly focus on methodological and theoretical concerns while in the second part I will focus on finding-policy disparities. In both parts, I will only focus on the tensions, as I see them emerging, without offering solutions, which I think smarter people should focus on.
A first anecdote for the first type of tension was demonstrated through the presentation of my longtime collaborator Doron Teichman from HUJI on anchoring legal standards (Teichman, Feldman & Schurr (R&R)) in a conference in Israel where both psychologists and legal scholars attended as audience. Anchoring in short is usually defined as ”a cognitive bias that describes the common human tendency to rely too heavily on the first piece of information offered when making a decision” (Shrotriya & Pandey, 2013). The classical studies in psychology (e.g. Tversky & Kahneman, 1974) which examined anchoring usually relied on the effect of some random four digits number or the number that emerged from a wheel of fortune on people’s judgment. In contrast, the studies done by legal scholars, including us, are usually related to damages asked by lawyers or numbers that emerged from a different legal source.
The mostly justified criticism of this argument is that legal usage of anchoring is not considered to be a pure anchoring effect because the original stimuli is not completely orthogonal to the target, meaning that the stimuli is not clean and might carry other types of rational influence. Clearly, the number argued for by a lawyer, even in an adversarial system, does have some meaning regarding the size of the claim, relative to the wheel of fortune example in the original psychological experiment.
That being said, we ask ourselves whether empirical legal studies need to compete with psychology on the same term. If we take the example of the wheel of fortune vs. the number coming up from a legal suit, clearly the first carries much greater internal validity – any kind of effect in the first case could not be interpreted as nothing but a cognitive bias. Admittedly, this is not the case when we look at the influence in judgment following a legal claim. However if we see empirical legal studies as an academic community which tends to understand how cognitive biases might affect the legal system, clearly, the likelihood that someone would use a wheel of fortune to try and affect a judgment is farfetched.
An additional area where the difference between ELS and psychology can be seen is related to methodological norms I see in psych journals but not in ELS scholarship. Among them, are norms related to ‘open science’ and a need to deposit in advance research hypotheses and planned number of experiments. In addition, I see requirement for replication which I don’t yet see in the same level in ELS scholarship, where sometimes it is acceptable to have a paper with one experiment (as in some economics papers). Moreover while this is changing, as of now, the level of measures on behavior, usually gathered in psychological journal are more extensive than those I usually (but not always) see in legal publications.
On the other hand, it seems that ELS has some areas that might suggest its methodology to be superior. For example, regarding the issue of the type of participants, I have noticed researchers get criticisms in ELS conferences which they would not get in psych conference. In addition, the expectation for multiple methods seems to be more common request by legal reviewers of grant proposal, in a way which you would not always expect in pure psych grants (although this is changing too). Along the same lines, the need to take into account various theories and alternative explanations, even from different disciplines, is not at the same level in psych research where it is more common to use only one type of theoretical school of thought. In the regard, the mere fact that in contrast to psychology or economics, ELS does not hold a clear agency model, has both positive and negative effects on the freedom of choosing methods which could be used to map a certain phenomenon (e.g. no firm expectations to use incentive compatible design).
Ideally, with the establishment of ELS in the years to come, we would see two positive developments: The first is to have a more cohesive ELS, where, say, researchers who do ELS from a behavioral perspective would be more integrated with researchers who do ELS from an institutional perspective. The second one is that it will be harder to accuse ELS of using less “sophisticated” methods, relative to the disciplines where it borrows its methods from.
Comments