Do Jurors Give Appropriate Weight to Forensic Identification Evidence?

/in

Emily C. Lennert

Category: Statistics

Keywords: juror, evidence, weight of evidence, Bayes, Bayesian expectation, likelihood, probability, false report probability, FRP, random match probability, RMP, DNA evidence

Article to be reviewed:
1. Thompson, W. C.; Kaasa, S. O.; Peterson, T. “Do jurors give appropriate weight to forensic identification evidence?” Journal of Empirical Legal Studies. 2013, 10 (2), 359–397.

Disclaimer: The opinions expressed in this review are an interpretation of the research presented in the article. These opinions are those of the summation author and do not necessarily represent the position of the University of Central Florida or of the authors of the original article.

Author’s note: Prior to reading this review, refer to the previous review, “Forensic Statistics 101: the Bayesian approach and common fallacies” for information regarding Bayesian statistical models, also known as the likelihood ratio approach.

Summary:

In 2009, the National Academies Press released a report on the state of forensic science, stating that error rates should be explored to estimate error rates, specifically for DNA analysis. The review article investigated the weight that jurors give to DNA evidence. Two statistics are commonly reported for DNA evidence, the random match probability (RMP) and false report probability (FRP). RMP is the probability that a randomly selected other person would match the genotype of the evidence. FRP is the probability that a match will be reported in an instance where no such match exists. This study focused on the effect of the reporting of FRPs on the way that jurors interpret evidence.

According to the authors, previous studies related to jurors interpreting statistical evidence reported that jurors appeared to give less weight to a forensic match than Bayesian models suggested they should (refer to the Author’s Note). However, the authors speculated that previous studies may have suppressed the juror’s estimates due to ceiling effects, meaning individuals’ tendencies to avoid extreme ends of a response scale. The authors state that changes in the way that statistics are presented can have a significant effect on the way that juror’s interpret evidence. FRPs introduce the possibility of a false positive fallacy. The false positive fallacy occurs when, based on misinterpretation of FRP, jurors overestimate the value of forensic evidence, relative to Bayesian norms. The false positive fallacy is more frequently observed in instances with low prior probability, also known as prior odds. This study sought to test whether this fallacy is made by jurors in a simulation, and how frequently.

Two experiments were created with the intent to test the impact of the FRP. The first experiment used a sample pool of 665 undergraduate students. Students were given instructions on how to use a probability scale for this experiment, then given a three page description of evidence from a hypothetical criminal trial. Three experimental conditions for FRP were given, but RMP was reported as 1 in 1 trillion to all for all experimental conditions. FRPs varied for each experimental condition: zero error, 1 in 10,000, and 1 in 100. This means that study participants were informed that the chance of a false match being reported were either zero, 1 in 10,000, or 1 in 100. The prior probability (refer to Author’s Note) of the non-DNA evidence varied in the experiments as well: low or high. Study participants answered a questionnaire regarding their thoughts on the defendant’s guilt as well as to check how well the participants understood the evidence and the statistics related. After statistically examining the results of the experiment, the authors determined that their efforts to manipulate the prior probability of the evidence affected the verdicts that participants chose. When prior probability was high, participants weighed DNA evidence appropriately compared to Bayesian norms. However, when prior probability was low, participants gave lower weight to the evidence than expected by Bayesian norms. Additionally, 75% of participants correctly recalled the RMP that was reported, and 7% of those who did not appeared to have remembered the number correctly but inverted the scale. The authors also reported that their efforts to manipulate the FRP also had an effect on the verdict that was delivered. Guilty verdicts were highest in the zero error condition, followed by the 1 in 10,000 condition, with the lowest amount of guilty verdict being delivered in the 1 in 100 condition. The authors state that this reflects the jurors’ understanding of the implications of FRPs. Although the authors intended to study the effect of FRP, experiment 1 studied two major variables: prior probability and FRP. Experiment 1 resulted in two major conclusions: prior probability has a statistically significant effect on outcome, and FRP also provides a statistically significant predictor of the outcome.

Experiment 2 was conducted on 305 participants from a local jury pool. The experimental parameters were changed slightly to adjust for what the authors described as “sources of ambiguity” in the first experiment. The prior probability was removed as a variable, such that all experimental parameters shared a low prior probability of guilt. This allowed experiment 2 to be focused solely on the effect of FRP, which the authors had stated was their goal. The FRP experimental conditions were maintained, although the description of the zero probability condition was redesigned slightly to make it more realistic. The authors found, after statistical evaluation of the results, that the 85.5% of participants correctly recalled the RMP. The authors reported that the majority of respondents in each FRP condition responded correctly for the condition in which they were placed.

The study concluded that, when participants were told that the FRP was high, they gave less weight to the evidence. Conversely, when participants were told that the FRP was low, they gave greater weight to the evidence. The authors conclude that study participants were found to make judgments consistent with Bayesian expectations, which does not correspond with previous research. The authors suggest that the scales used in this experiment, which allowed participants to give judgments higher than Bayesian norms but not at the extreme of the scale, avoided the ceiling effect that was believed to have occurred in previous studies.

Scientific Highlights:

  • False report probability impacts the way a jury interprets evidence.
  • When FRP is low, jurors give greater weight to evidence, consistent with Bayesian expectations.
  • When FRP is high, jurors give less weight to evidence, consistent with Bayesian expectations.
  • Both experiments yielded similar results.

Relevance: Contrary to the results of previous research, this study found that jurors appear to interpret DNA evidence properly. The correct weight was given to the evidence, considering the prior probability and false report probability.

Potential conclusions:

  • Jurors appear to give correct weight to DNA evidence, consistent with Bayesian expectations.
  • Proper instruction on how to interpret the statistical data, as well as proper reporting of the FRP and RMP can impact the weight given to the evidence.
  • When FRP is high, (i.e. 1 in 100) evidence is seen as less reliable and less weight is given to the evidence. However, when FRP is low, (i.e. 1 in 1,000,000) evidence is seen as more reliable and greater weight is given to the evidence.