Labs must correct wrong DNA mixture analyses, learn when not to analyze 'crap'

Yesterday for work I attended a Forensic Science Commission committee meeting in Dallas on DNA mixtures where the agenda had suggested they'd be parsing prosecutor disclosure obligations and mapping out a path toward reviewing old cases. Instead, the committee couldn't field a quorum, so four scientists brought in to advise them were left to field a lengthy panel-discussion/Q&A which clarified some issues and on others, only emphasized how muddy much of this remains.

Terri Langford at the Texas Tribune was the only reporter there, here's her story. In general, she correctly summarized:

experts tried to temper the expectations about DNA testing that were built over more than a decade.
 
"One of the problems was DNA was called the gold standard," Bruce Budowle, director of the University of Texas Health Science Center's Institute of Applied Genetics, said. "Big mistake."

Budowle said DNA deserved gold-standard status when it came to a single DNA sample compared to a single suspect, or even in rape kits where there are two samples and one (the woman's) is known. But when analyzing mixed DNA samples where no one is definitively known, or even where labs can't tell precisely how many DNA contributors there are, analysts engage in interpretation which has not always been informed by best practices. Cutting-edge science takes too many years to trickle down from the research labs to the crime-lab work bench, the panelists repeatedly emphasized.

Budowle said the 2009 National Academy of Sciences report "gave DNA a pass" and it shouldn't have - interpretation of DNA mixtures has a subjective human element just like other comparative forensics.

We learned a bit more about how all this came up: When Galveston DA Jack Roady asked for DNA results to be reinterpreted in one of his cases, the probability the DNA matched their defendant went from more than one in a billion to one in 38.

But that was the FBI lab, having already corrected their method.* Yesterday we learned more about recent changes in DPS' DNA mixture interpretations. Again, from Langford:

Crime labs have recently adopted the new “mixed DNA” standard. The DPS switched to it on Aug. 10. The move has prompted prosecutors like [Inger] Chandler to resend evidence in pending cases to the lab to have the data analyzed using the new standard. In Houston's Harris County, that's about 500 pending cases where DNA evidence will be introduced at trial.

In addition, DAs are notifying defendants who are already convicted about the new standard. For example, Harris County prosecutors have already notified those convicted of capital murder and awaiting execution. It is not known how many of the 253 inmates on Texas’ death row were convicted with mixed DNA. Of the 253 inmates on Texas death row, 90 are from Harris County.

The new standard at DPS deserves further elaboration because the expert panelists universally agreed that the old method was wrong and improperly interpreted results.

After yesterday, I understood for the first time (perhaps it was said before and didn't penetrate my notes/consciousness/thick skull) that DPS' DNA labs had not changed their protocols until this issue came up while your correspondent was on vacation last month. And the details of the change were significant.

First, a bit of background. DNA testing looks at two metrics on X and Y axes: Whether alleles are present at various loci, and the quantity of DNA available for testing at that spot. (The latter is complicated by allele drop-in, drop-out, and stacking, terms I'm only beginning to understand.) When examining the peak height of DNA quantity on the test results, DPS' old method did not impose a "stochastic" threshold, which as near as I can tell is akin to the mathematical sin of interpreting a poll without ensuring a random sample.  (The word "stochastic" was tossed around blithely as though everyone knew what it meant.) Basically, DPS did not discard data which did not appear in sufficient quantity; their new threshold is more than triple the old one.

That new methodology could change probability ratios for quite a few other cases, the panel predicted. One expert showed slides demonstrating how four different calculation methods could generate wildly different results, to my mind calling into question how accurate any of them are if they're all considered valid. Applying the stochastic threshold in one real-world case which he included as an example reduced the probability of a match from one in 1.40 x 10⁹ to one in 38.6. You can see where a jury might view those numbers differently.

Not every calculation will change that much and some will change in the other direction. The application of an improper statistical method generates all types of error, not just those which benefit defendants. There may be folks who were excluded that become undetermined, or undetermined samples may become suspects when they're recalculated. The panel seemed to doubt there were examples where a positive association would flip all the way to excluded, but acknowledged it was mathematically possible.

DPS has identified nearly 25,000 cases where they've analyzed DNA mixtures. Since they typically represent about half the state's caseload, it was estimated, the total statewide may be double that when it's all said and done. Not all of those are problematic and in some cases the evidence wasn't used in court. But somebody has to check. Ch. 64 of the Code of Criminal Procedure grants a right to counsel for purposes of seeking a DNA test, including when, "although previously subjected to DNA testing, [the evidence] can be subjected to testing with newer testing techniques that provide a reasonable likelihood of results that are more accurate and probative than the results of the previous test." So there's a certain inevitability about the need to recalculate those numbers.

Making the situation even more complex, next year DPS will abandon the updated method and shift to "probabilistic genotyping," which has the benefit of using more of the DNA data but asks a mathematically different question than the old method. Instead of calculating how many people in the population share DNA traits with the sample, the new method calculates, e.g., how likely it is that two patterns match the suspects compared to any other two random people.

That's a subtle difference, but it means the new DPS method is not a direct refutation of the old one, prosecutors exasperatedly realized upon questioning the panel. Going forward, it's probably best to shift to probabilistic genotyping until something else comes along, they were told. For older cases, though,labs would probably need to calculate both. That stickies the wicket quite a bit - they can't just wait and issue results under the new method in old cases, as some labs had been advising. They'll have to recalculate them using the new stochastic threshold.

Another interesting side note: the old method always generates the same result. Because of statistical modeling, probabilistic genotyping will get a different result every time (presumably within a valid range of error). That made me wonder about the wisdom of moving to a system where results are not entirely replicable. That's an issue for the courts, one supposes, which will ultimately need to decide which approach they prefer. All this will end up before the Texas Court of Criminal Appeals sooner than later, most observers agreed.

Even when labs shift to a new method, though, the software implementing these models cannot be treated as a black box, the panel emphasized. There's inherently an interpretation element and without understanding the different statistical methodologies, they warned, crime labs could still get into trouble, a likelihood which became increasingly apparent as the hours-long session progressed. "All models are wrong but some are useful," one panelist quipped. Each are a different tool and one uses different tools for different things.

One final takeaway: Labs not only need to update their methods for performing statistical calculations, just as importantly they need to create standards for when they should make no calculation at all. One panelist gave an anecdote from a 2013 study: 108 labs were given a sample he'd created using four DNA sources, but for context he told them the names of three people, only two of whom were actual sources. Amazingly, 75 percent of labs mistakenly said the sample came from three people and included the person who wasn't a source. Only 20 percent said they couldn't make a calculation. If that's not a red flag, I don't know what is!

Budowle, who for 26 years worked for the FBI and was their lead expert on these topics, said that when there are too many DNA sources to make an interpretation, as is increasingly the case with touch DNA samples, the scientific term for what one sees in the test results is "crap." They'd operated in the past on the assumption that examiners could recognize crap, he said, but it's becoming apparent guidance needs to be developed because people are busily applying these statistical  models in invalid and problematic ways. All the other panelists agreed.

Finally, everyone agreed, this is not at all  just a Texas issue but is a national and even international problem. Everywhere DNA analysis is used for crime fighting, courts and labs eventually must grapple with these issues, and many jurisdictions have yet to do so. Texas crime labs weren't acting in bad faith on this; this isn't a drama with a villain. As science advanced, past errors became known, it's nothing nefarious, however problematic it may be for the justice system to have replied on unproven science. Texas is just confronting the issue first in large part because of leadership from the Forensic Science Commission. Their executive director Lynn Garcia has ably pieced together stakeholders and generated a meaningful, high-level conversation among decision makers, even if few decisions have been made yet.

The committee will meet again before the next Forensic Science Commission meeting Oct. 2, perhaps the day before, to take up the agenda they didn't get to yesterday in Dallas. Fascinating stuff. What a mess!

* CORRECTION: A commenter correctly noted Roady's sample was retested by DPS, not the FBI. See here.