Review: Error and its Meaning in Forensic Science

Emily C. Lennert

 

Category

Statistics

Keywords

error, rate, mistake, uncertainty, limitation, Daubert, admissibility, proficiency

Article Reviewed

1. Christensen, A. M.; Crowder, C. M.; Ousley, S. D.; Houck, M. M. Error and its meaning in forensic science. Journal of Forensic Sciences. 2014, 59 (1), 123-126.

Additional Reference

1. Strengthening forensic science in the United States: a path forward; National Academies Press: Washington, D.C., 2009.

Disclaimer

The opinions expressed in this review are an interpretation of the research presented in the article. These opinions are those of the summation author and do not necessarily represent the position of the University of Central Florida or of the authors of the original article.

Summary

The topic of “error” as it pertains to forensic science has been of particular importance since the Daubert ruling in 1993 and the National Academy of Sciences’ (NAS) National Research Council report, “Strengthening Forensic Science in the United States: A Path Forward”, published in 2009.

The Daubert decision redefined admissibility criteria for expert testimony, providing guidelines to ensure dependable and valid scientific or technical expert testimony.

Years later, the NAS report highlighted scientific and technical challenges being faced in the forensic science community. The report expressed concerns that some disciplines lacked scientific rigor, noted a need for “more and better research”, and provided several recommendations to improve the state of forensic science.2 Among the recommendations provided in the report, recommendation three states “Research is needed to address issues of accuracy, reliability, and validity in the forensic science disciplines.”2

In the Daubert decision, “reliability” is discussed repeatedly. Scientifically, the term “reliability” is generally used to express the degree of variability in observations between different observers, and includes how well the technique can be repeated. For example, if 100 examiners were given a sample to examine by a particular technique, and 99 examiners reached the same conclusion, the technique would be regarded as highly reliable. In the same scenario, if only 45 examiners reached the same conclusion, and the conclusions of the remaining 55 examiners varied, the technique would have low reliability. Reliability establishes how well a method can be repeated, but does not necessarily mean that the method will produce correct conclusions. The authors note that the use of “reliability” in the Daubert case appears to be misused, and that “dependability” is what the court intended. Dependability, scientifically speaking, encompasses reliability as well as validity. Reliability alone cannot establish validity. Validity is most simply defined as the overall odds of reaching a correct conclusion by the given technique. A method that produces correct conclusions more often than what is considered random chance, i.e. a high probability of correctness, will be considered valid. This concept is why method validation studies are necessary for novel techniques and applications in science. Validity of a method is an important aspect of establishing admissibility.

What is “Error”?

A known or estimated error rate is a measure that can be used to represent the validity of a method. Methods with high error rates will be considered to have low validity, and low error rates will indicate higher validity. Several sources of error may exist, which may impact the error rate of a method.

Practitioner Error

Practitioner errors are mistakes and operator errors. Practitioner errors are generally accidental, but may be intentional, in cases of fraud. The errors are primarily accidental, and may originate from random errors, systematic problems, negligence, or incompetence. Errors may include transposition of numbers while recording data, incorrect use of instrumentation, improper selection of analysis methods, improper application of the method for analysis, etc. Less commonly, error results from fraudulent behavior, such as in many cases examined by former Massachusetts state chemist Annie Dookhan. Practitioner error is difficult to estimate, but may be reduced through quality assurance systems, training, proficiency exams, peer review, and following validated protocols and best practices within the discipline. Regardless of the reason for the error, practitioner error is human error and, therefore, not considered scientific error. Practitioner error is not a factor in determination of error rate for a method.

Instrument/Technological Error

Instrument error is defined by the difference between a measured value indicated by the instrument and the actual value, i.e. ground truth. Instruments should be calibrated against a reference material, but after calibration most have an acceptable amount of error. For example, if a cocaine sample is analyzed via direct analysis in real time – mass spectrometry (DART-MS), one would expect to see a protonated cocaine peak at 304.1549 m/z. If the observed peak was 304.1548 m/z, this would not be an error because the measurement falls within the acceptable error range. However, if 304.1501 m/z was observed, this measurement would be outside the acceptable range and would be an instrumental error. Proper maintenance and calibration serve to minimize instrument error. However, an acceptable amount of error is recognized.

Statistical Error

Statistical error is the deviation between the actual, i.e. ground truth, and predicted, i.e. calculated, values. Statistical error generally represents normal variability that is characteristic of measurements and estimates; it is typically represented by standard error or other measures of uncertainty.

Method/Technique Error

Method error is the error that is due to inherent limitations of a given method, not relating to practitioner or instrument error. Generally, method error is due to limitations such as overlap of measurements or traits among different groups, or the frequency of an observed trait within the population. These limitations are not errors, technically speaking; however, they do affect the method’s sensitivity, resolving power, probative value, and ultimately impact the validity of the method. For example, in the determination of sex by skeletal remains, the human pelvis possesses greater resolving power compared to the skull. Greater overlap exists between male and female skull characteristics as compared to pelvic characteristics, i.e. greater method error exists in using the skull vs the pelvis, making the pelvis more discriminatory in the determination of sex. Method error estimations are generally the most familiar of all errors. Error of a method cannot be minimized, since it originates from inherent limitations of the method itself, but better methods can be developed.

Commonly, in the determination of admissibility, error rates are determined primarily from statistical and method error. Well researched methods with proper method design and appropriate statistical models can serve to provide valid and reliable scientific methods.

Common Misconceptions in Understanding Error

If error is misunderstood or misrepresented, it can have serious legal and scientific consequences. Reasons for misunderstanding or misrepresentation vary. Error and error rates in a scientific sense may simply not be understood. Additionally, lack of proper statistical training or a lack of understanding in the process of science may be a source of error misrepresentation and misunderstanding. The authors note that error rates may also be misrepresented due to concerns about the given method being exposed as lacking a strong scientific backing. Sources of error misuse are discussed below.

The “Zero Error Rate” Claim

To discuss the zero error fallacy, the authors cite expert testimony from People v Gomez (99CF 0391, 2002). A fingerprint examiner testified to a zero error rate, stating “And the reason we make that bold statement is because we know based on 100 years of research that everybody’s fingerprints are unique, and in nature it is never going to repeat itself again.” The first issue with this statement is that the examiner fails to acknowledge that, scientifically, the concept of uniqueness is probabilistic and impossible to prove with absolute certainty. Second, the examiner fails to recognize other contributors to error. Despite the low probability of identical fingerprints originating from different individuals, error or limitations may still be present in the methodology used for comparison. Error rate is not determined solely by how frequently the trait, e.g the individual fingerprint, is observed in a population. The accuracy of the method used impacts error rate. Granted, in disciplines where the method of analysis is primarily examiner observation, such as fingerprint comparison, it may be impossible to discern method error from practitioner error. Regardless, to claim a zero error rate is incorrect, and the error rate is a non-zero probability.

The “No Error Rate Can Be Determined” Claim

Some practitioners claim that error rates cannot be determined for methods. The authors cite a lack of proper testing to determine method limitations or potential error as probable causes for this claim. Insufficient statistical training or misunderstanding of error may also lead to this claim being made. The authors argue that, “if a method can be applied, error may exist and should be acknowledged.”

Miscalculation of Error Rates

In cases where a known or potential error rate has not been determined by method validation, error rates may be incorrectly derived from other sources. Proficiency tests, for example, are designed to monitor performance, demonstrate reliability in the work produced by analysts, and demonstrate that procedures are being properly followed. These tests are not designed to accurately determine error rates; however, some individuals will attempt to extrapolate error rates from proficiency testing data. These exercises are designed to evaluate practitioners’ ability to perform analyses, and are therefore used to identify and prevent practitioner error. Errors reflected in proficiency tests are not representative of real error rates, which are dependent primarily on statistical and method error. Similarly, previous studies may be used to extrapolate error rates. However, if the nature of the study was not to determine error rates, then error rates should not be extrapolated. To use error rates from a study properly, the study should be designed specifically to determine statistical and methodological error rates.

Error is an important factor in the determination of method validity. The validity of a method must be considered when admissibility is under consideration. Therefore, it is crucial to understand the different types of error that exist and which errors contribute most to validity and error rate determination. Additionally, it is imperative that common misconceptions of error are understood and avoided, so that error may be interpreted and implemented properly in both scientific and legal contexts.

Highlights

• Dependability encompasses reliability and validity, both of which are key factors in determining admissibility by Daubert criteria.
• Practitioner error is a source of human error, and differs from scientific error.
• Scientific error includes instrumental, statistical, and method error.
• Statistical and method error are primary contributors in the determination of error rate.
• Several misconceptions exist in understanding error. It is crucial to the proper implementation of error rates that these misconceptions are understood and avoided.

Relevance

While error is a vital part in the determination of validity, which impacts admissibility by Daubert criteria, the concept is often misunderstood. Understanding the sources of error and avoiding misconceptions in error are vital to proper use of error rates in admissibility decisions.