Digesting new Daubert opinion on DNA mixtures

Grits earlier referenced the opinion from federal District Judge Janet Neff, in Michigan's western district, who directly evaluated the most common, commercial DNA-mixture software using probabilistic genotyping under the Daubert standard for whether to allow expert testimony. Texas courts heretofore have let in such evidence but avoided a direct Daubert evaluation.

There was a Texas connection in the case. Dr. Michael Coble, of the University of North Texas Health Science Center, was the expert employed by the prosecution. He has been one of several key advisers to the Texas Forensic Science Commission and the legal system in general as the state has struggled with this DNA-mixture issue. The expert for the defense, whose arguments more frequently prevailed in Neff's opinion, was Dr. Dan Krane, from Wright State in Ohio.

Judge Neff evaluated each prong of the Daubert standard as it related to the STR-Mix DNA mixture-analysis software. This program employs high-end math that few DNA-lab practitioners can actually reproduce called "probabilistic genotyping." But programs are only as good as the assumptions that guide their work, and some of the assumptions in each case - e.g., the number of contributing samples - are inputted by the user. There are subjective judgment calls throughout the process. Further, because the math results from tens of thousands of randomized guesses, the results aren't replicable test to test.

Judge Neff was disquieted that, unlike for one-to-one DNA matching or two-source mixtures, as with a rape kit, there are no controlling standards for interpreting DNA mixtures, especially for samples with very small amounts of DNA - maybe only a few scattered cells.

The judge also made the interesting point that, even if the math behind the STR-Mix software is impeccable, there were international standards for coding such high-end math projects and the STR-Mix software didn't meet them all. Coble was dismissive that other fields might have standards to contribute, but the product in question is so obviously an intersection of an array of multidisciplinary work, the judge didn't buy it.

Further, the judge echoed concern that most validation studies on the software so far have been conducted by company principals. And while there was peer-reviewed support for STR-Mix (again, mostly for studies by people with a financial stake in the project), none of that peer-reviewed work focused on low-level samples as in this case, where just a few cells were found that were said to belong to the defendant.

One point that's always bugged me was raised in an expository section of the opinion but not included in her reasons for her Daubert denial: “[D]ue to the methodologies used in STRMix, '[t]he results of no two analyses will be completely the same',” she wrote. Elsewhere, she quoted an academic article noting that, "for a technique to be broadly applicable, it must be based on measurements that can be replicated." Supporters of the software, however, say replicability isn't necessary and it should be good enough if the results were all in some acceptable range. Regardless, while Judge Neff clearly observed this incongruity, it wasn't included in her final reasons for her ruling.

Clearly the most persuasive piece of evidence for Judge Neff was the 2016 PCAST report which expressly stated that STR-Mix was validated for up to three sources if the sample to be identified is at least 20 percent of the total. She did not find further validation regarding smaller samples, and the very small sample in this case, she believed, merited even greater caution. She essentially treated the threshold cited by PCAST as a bright line regarding the current state of the science. STR-Mix can be used for traditional one or two-source DNA matching, she ruled, and for three-source mixtures if the sample to be matched made up 20 percent or more of the whole. But otherwise, the estimates would be inadmissible.

One small upside for backlogged crime labs using STR-Mix: This would make it easier to screen DNA mixture submissions. Samples with more than 3 contributors should probably be rejected up front as unanalyzable "crap," to use the words of a prominent DNA expert who advised Texas crime labs on the topic.

Errors in such cases can cause false convictions, so Grits is glad to see this increasingly dirt-common evidence more carefully vetted. From the time probabilistic genotyping was first introduced to Texas crime labs in 2015, experts were warning against stretching the limits of the math (i.e., trying to analyze "crap"). Judge Neff would set a bright-line threshold, and given the current state of the literature, it's not an unreasonable choice. In many ways, it's the most defensible (it would probably be easier to get past it by improving the tech than defeating her arguments). But whatever threshold one chooses, her decision highlights that one must exist, even if courts haven't heretofore required it.

I'd quoted it earlier, but I loved Judge Neff's conclusory analysis of current DNA-mixture-evidence practices, so let's give her the final word here:

The DNA evidence sought to be admitted in this case—in essence, that it is 49 million times more likely if Daniel Gissantaner is a contributor to the DNA on the gun than if he is not— is not really evidence at all. It is a combination of forensic DNA techniques, mathematical theory, statistical methods (including Monte Carlo-Markov Chain modeling, as in the Monte Carlo gambling venue), decisional theory, computer algorithms, interpretation, and subjective opinions that cannot in the circumstances of this case be said to be a reliable sum of its parts. Our system of justice requires more.

Anyone interested should definitely read the whole opinion. This summary hardly does it justice.

MORE: From Techdirt.

See prior, related Grits posts:

• Most crime labs accused innocent person in DNA mixture study
• A reluctant scoop: Changing interpretations of DNA mixtures vex legal system
• Labs must correct wrong DNA mixture analyses, learn when to recognize 'crap'
• Resources on DNA mixtures
• The challenge of notifying defendants in large-scale forensic error cases
• Courts punt on forensics surrounding DNA mixtures
• Black box DNA mixture analysis method used in NYC turned out to be flawed
• Changes to DNA mixture software raise questions about old versions
• News flash: 'Touch DNA' doesn't necessarily require touching
• DNA mixture SNAFU a mess but don't expect 'deluge' of innocence claims