Review: Forensic Statistics 101: The Bayesian Approach and Common Fallacies
Emily C. Lennert, Candice Bridge, Ph.D.
Category: Forensic Statistics
Keywords: Statistics, Bayes, Bayesian, probability, evidential value, likelihood ratio, prosecutor’s fallacy, defense fallacy
Article to be reviewed:
1. Sjerps, M. F”orensic statistics.” Nieuw Archief Voor Wiskunde. 2008, 5/9(2), 150-155.
Additional references:
2. Thompson, W. C.; Schumann, E. L. “Interpretation of statistical evidence in criminal trials: The prosecutor’s fallacy and the defense attorney’s fallacy.” Law and Human Behavior. 1987, 11 (3), 167–187.
Disclaimer: The opinions expressed in this review are an interpretation of the research presented in the article. These opinions are those of the summation author and do not necessarily represent the position of the University of Central Florida or of the authors of the original article.
Summary: Statistics, although not often thought of in the context of the law, play an important role in the judicial system. Probability theory is directly connected to the judicial process. Generally, each case has two competing hypotheses: the hypothesis presented by the prosecution (Hp) vs the hypothesis presented by the defense (Hd). In court, an expert witness will often give statements regarding the strength of evidence that can be underwritten by statistics and probability theory. The outcome of a case depends on which hypothesis is determined to be the most probable, based on the evidence presented and the prior probability. The prior probability (i.e. prior odds) is associated with the probability of the event in question occurring based on information known before forensic evidence in introduced or before the cases even begins. Examples include: the defendant was at the crime scene when the crime was committed, a white man was seen leaving scene when the crime occurred, or the defendant’s car was spotted driving in the area of the crime. This application of statistics to forensic science and the criminal judicial system is referred to as forensic statistics.
Forensic statistics follows a Bayesian approach, also known as the likelihood ratio approach, which is based off of the well-known probability theory by Bayes.1 The Bayesian approach allows a forensic investigator to determine evidential value, i.e., the strength of the evidence, and report the evidential value in the form of a likelihood ratio. A likelihood ratio is a quantitative, or numerical, method for evaluating the strength of evidence. A forensic investigator serving as an expert witness should not report any findings about the prior odds; they should only report on the strength of the evidence. They can discuss the likelihood of the evidence being present at the crime scene given the prosecution’s hypothesis against the likelihood of the evidence being present at the crime scene given the defense’s hypothesis. This results in the likelihood ratio of that evidence, which can be calculated by dividing Hp by Hd (Hp/Hd). This means that the forensic investigator makes statements strictly regarding the value of the evidence by presenting this likelihood ratio. Then, the judge or jury will consider the likelihood ratio based on the evidence and the prior odds to determine the posterior, or final, odds, and base the verdict upon the posterior probability ratio. The posterior probability (i.e. posterior odds) will multiply the prior odds by the likelihood ratio of the evidence to come to a final number which may lean more towards the prosecution’s theory or more towards the defense’s theory. The Bayesian approach, when implemented properly, should result in an unbiased presentation of the evidence to the judge and jury and provide a rational method for the judge and jury to combine evidence and draw conclusions. The Bayesian approach to forensic statistics is often presented in the form of an equation:
The posterior, or final, odds.
This is the ratio of the probability of each hypothesis, given the evidence, which is what a judge or jury will use to reach a verdict.
The Bayesian method has helped to uncover three important fallacies in the interpretation of evidence.1 These fallacies lead to improper implementation of evidence and can lead to wrongful acquittal or wrongful conviction. The prosecutor’s fallacy occurs when a conclusion is drawn based on the match probability of the evidence without considering the full likelihood ratio or prior odds.2 For example, a partial DNA profile from a crime scene is matched to a defendant’s DNA profile, and the probability of a DNA profile match with any random person is given as one in one million. The prosecutor may incorrectly conclude that this means there is a one in one million chance that the crime scene DNA is not the defendant’s. This conclusion is incorrect because the prosecutor does not consider the prior odds. The defense fallacy occurs when the assumption is made that the prior odds for the defendant having committed a crime is the same as the prior odds of any person within a population committing the crime.1, 2 Following the previous example, the crime occurred on an island with a population of approximately 20 million; given the match probability of one in one million, twenty people, including the defendant, will match the DNA profile. The defense may conclude that there is a one in twenty chance, 5%, that the defendant is the individual who deposited the DNA at the crime scene. This conclusion is incorrect because it makes the assumption that all individuals who match the profile, which may include infants, disable persons, etc., had equal opportunity to commit the crime. The final fallacy is the base rate fallacy, where the likelihood ratio is not scaled by the prior odds.1 For example, the likelihood for the evidence being present given the prosecution’s hypothesis is given as one in ten, while the likelihood for the evidence being present given the defense’s hypothesis is given as one in one thousand, and the resulting likelihood ratio value is 100.
This equation is calculated by:
The likelihood ratio.
This ratio shows the value of the evidence introduced, indicating which hypothesis the evidence favors. This ratio should be given by the expert witness.
The prior odds.
This is the ratio of probabilities prior to the introduction of evidence, either at the beginning of the trial or at the introduction of new evidence.
By stating that the probability of the prosecution’s hypothesis is 100 times higher than the defense’s, the base rate fallacy is committed. The prior odds of the event, also referred to as the base rate frequency of an event, were not considered in this equation only the likelihood of the evidence.1,2 These fallacies may be avoided with the help of both lawyers and expert witnesses. An expert witness must be mindful of the way he or she presents evidence and conclusions in court. Poor phrasing of the information may lend to the commission of one of these fallacies. Lawyers can help to prevent the fallacies by becoming more educated in forensic statistics and the interpretation of evidence and application of likelihood ratios.
Relevance: The implementation of forensic statistics can reduce the implicit or undue bias that can occur if the weight of the evidence is not understood, which can influence the outcomes of judicial processes. Understanding the theories and methods behind the statistics involved is helpful to proper interpretation of evidence and expert witness testimony.
Potential conclusions: An expert witness should only provide statements that attest to the value of the evidence, such as the presentation of likelihood ratios of evidence. It is the job of the judge and jury to integrate the weight of the evidence and the prior odds to determine the probability of guilt or innocence (i.e. posterior odds). Attorneys may help to ensure proper integration by avoiding three major fallacies that are often found in the interpretation of evidence. By following the Bayesian approach properly, convictions or acquittals may be made through a logical approach, avoiding wrongful convictions or acquittals.
Legal Brief: Forensic Statistics 101: the Bayesian approach and common fallacies
Steve Krejci
The use of statistics in forensic testimony is currently the subject of a pending update to the 2009 NAS report. Still open for comment as of this writing, the National Committee on Forensic Science generally counsels against the use of statistical terms when the underlying data has no basis in statistics. The Bayesian approach, when applied to weight of given evidence, provides a useful concept for explaining weight, but experts should stop short of any such comments on the weight. Definitive statements, such as “It’s the defendant’s fingerprint,” should be avoided. Rather, the strength of the forensic evidence or its weaknesses should be explained to the jury. Any uncertainties associated with a given statistic should also be reported. National Commission on Forensic Science, Views of the Commission: Statistical Statements in Forensic Testimony, Aug. 22, 2016 last accessed at https://www.regulations.gov/document?D=DOJ-LA-2016-0018-0005 on Sept. 26, 2016.
