Black-box DNA mixture analysis method used in NYC turned out to be flawed

Today the New York Times and ProPublica published a joint investigation by Lauren Kirchner into the use of DNA mixture evidence in New York titled "Thousands of Criminal Cases in New York Relied on Disputed DNA Testing Techniques."

The story covered some of the same ground as did a recent segment in Just Liberty's Reasonably Suspicious podcast discussing a DNA mixture case out of the Tyler 12th Court of Appeals here in Texas.

The big news out of the ProPublica story: After years of keeping its DNA-mixture analysis algorithms secret, the New York City lab finally had to reveal its source code (aka, the "FST code") to the defense and it didn't pass muster:

Again, the government refused to hand it over on the grounds that it was a “proprietary and copyrighted” statistical tool owned by the City of New York. 

The federal judge granted the defense access to the FST code in June 2016 under an order that bars wider disclosure. (The medical examiner’s office denied ProPublica’s public records request for the code, citing its “sensitive nature.”) 

Nathaniel Adams, a computer scientist and an engineer at a private forensics consulting firm in Ohio, reviewed the code for the defense. He found that the program dropped valuable data from its calculations, in ways that users wouldn’t necessarily be aware of, but that could unpredictably affect the likelihood assigned to the defendant’s DNA being in the mixture. 

“I did not leave with the impression that FST was developed by an experienced software development team,” Adams wrote in an affidavit. Pending more rigorous testing, “the correctness of the behavior of the FST software should be seriously questioned.” Characterizing Adams’ criticisms as merely stylistic rather than substantive, the lab told ProPublica that FST provided reliable calculations. 

Technology consultants wrote the software code for FST, according to a spokeswoman at the medical examiner’s office. Few, if anyone, at the lab or on the state’s DNA Subcommittee had the expertise to double-check the software, said a scientist in the lab who worked on the techniques who asked to remain anonymous for fear of career repercussions. “We don’t know what’s going on in that black box, and that is a legitimate question,” the scientist said, adding that evidence in older cases should “absolutely” be retested in light of growing questions about FST. “As a scientist, I can’t say no.”

The issue of proprietary "black box" software interpreting DNA mixture evidence mirrors the situation with private vendors used in Texas like STR-Mix and TrueAllele, as discussed on the Reasonably Suspicious podcast.

When New York's proprietary software was picked apart by the defense, they found previously unrevealed errors and flaws. Who's to say that won't also be the case with proprietary private software? The likelihood surely is greater than not.

Courts should allow defense experts access to the source code if they're going to admit DNA mixture evidence from these black box systems. There's just too much recent error and loads of evidence that the math and science involved in these black-box interpretations are well beyond the ken of most analysts seeking to interpret them.

Here are a few other tidbits from the ProPublica/Times story which add new information to the discussion here in Texas:

The FBI in May claimed to have validated STR-Mix for use in DNA mixtures involving up to five people. But having a law enforcement agency "validate" the method is far different from having validation from independent sources which would satisfy defense experts. In one famous case out of upstate New York, STR-Mix and their main competitor, TrueAllele, tested the same evidence and came back with different results.

"In the past three years, flaws in DNA methods have temporarily shut down testing in public crime labs in Austin, Texas, and Washington, D.C."

“Lab analysts 'make it seem like it’s a completely objective process,' said Bicka Barlow, a lawyer in California with a master’s degree in genetics and molecular biology. 'But I’m 100 percent convinced that there are many people who are incarcerated who were convicted with DNA evidence who are innocent.'”