Reporters mustn't overstate forensics accuracy - ballistics edition

In light of ballistics evidence from a federal database leading to arrests in linked, gun-related crimes, the Houston Chronicle last week ran a feature on how an ATF initiative is using ballistics evidence compiled in a national database to solve gun crimes.

But the story failed to acknowledge shortcomings with ballistics evidence and overstated the accuracy of firearms matching. In particular, they repeated the claim by law enforcement that, "The ATF database allows firearms experts to match high-resolution photos of marks left on bullet casings after being fired. The guns' firing pins leave a mark unique to each gun, allowing investigators to connect casings fired at different shootings."

The "unique" part is unproven. And the phrase "match" is overstated. The seminal critique on this topic came from the 2009 National Academy of Sciences (NAS) report, "Strengthening Forensic Science: A Path Forward." Ballistics matching is one of the areas of non-scientific "forensic science" being challenged in the innocence-era wave of re-evaluation.

The NAS report specifically discussed the National Integrated Ballistics Information Network (NIBIN) database described in the Chronicle article. At the time, it was still under development. However, the NAS cautioned that, in the end, the computer still doesn't do the matching. That's in part because, as they also showed, there are no standards to match to; it's a subjective comparison performed by an examiner. With the NIBIN system, according to the NAS report:

the final determination of a match is always done through direct physical comparison of the evidence by a firearms examiner, not the computer analysis of images. The growth of these databases also permits examiners to become more familiar with similarities in striation patterns made by different firearms. Newer imaging techniques assess toolmarks using three-dimensional surface measurement data, taking into account the depth of the marks. But even with more training and experience using newer techniques, the decision of the toolmark examiner remains a subjective decision based on unarticulated standards and no statistical foundation for estimation of error rates. The National Academies report, Ballistic Imaging, while not claiming to be a definitive study on firearms identification, observed that, “The validity of the fundamental assumptions of uniqueness and reproducibility of firearms-related toolmarks has not yet been fully demonstrated.”

So there is no computer matching of ballistics, it's done by examiners who make a "subjective decision based on unarticulated standards and no statistical information about error rates." Knowing that, all of a sudden, accusations based on ballistics evidence may appear less certain than portrayed in the story by the ATF.

For more background on critiques of ballistics and toolmark evidence (the former discipline is a subset of the latter), check out the 2009 NAS report. (Ctrl F and search on "toolmark" to find the relevant passages.) That report changed the terms of debates regarding traditional forensics like ballistics. And even though it's now nearly ten years old, with few exceptions (arson, hair and fiber analysis), forensics fields have taken at most baby steps to address the problems.

IMO, reporters should no longer write uncritically about disputed, comparison-based forensic evidence - even when police say it led to a "match" and make arrests based on the findings - without acknowledging that these are subjective comparisons, not scientific results. We've seen too many innocence cases featuring flawed forensics for the system to project that level of certainty.