Tuesday, September 05, 2017

Black-box DNA mixture analysis method used in NYC turned out to be flawed

Today the New York Times and ProPublica published a joint investigation by Lauren Kirchner into the use of DNA mixture evidence in New York titled "Thousands of Criminal Cases in New York Relied on Disputed DNA Testing Techniques."

The story covered some of the same ground as did a recent segment in Just Liberty's Reasonably Suspicious podcast discussing a DNA mixture case out of the Tyler 12th Court of Appeals here in Texas.

The big news out of the ProPublica story: After years of keeping its DNA-mixture analysis algorithms secret, the New York City lab finally had to reveal its source code (aka, the "FST code") to the defense and it didn't pass muster:
Again, the government refused to hand it over on the grounds that it was a “proprietary and copyrighted” statistical tool owned by the City of New York. 
The federal judge granted the defense access to the FST code in June 2016 under an order that bars wider disclosure. (The medical examiner’s office denied ProPublica’s public records request for the code, citing its “sensitive nature.”) 
Nathaniel Adams, a computer scientist and an engineer at a private forensics consulting firm in Ohio, reviewed the code for the defense. He found that the program dropped valuable data from its calculations, in ways that users wouldn’t necessarily be aware of, but that could unpredictably affect the likelihood assigned to the defendant’s DNA being in the mixture. 
“I did not leave with the impression that FST was developed by an experienced software development team,” Adams wrote in an affidavit. Pending more rigorous testing, “the correctness of the behavior of the FST software should be seriously questioned.” Characterizing Adams’ criticisms as merely stylistic rather than substantive, the lab told ProPublica that FST provided reliable calculations. 
Technology consultants wrote the software code for FST, according to a spokeswoman at the medical examiner’s office. Few, if anyone, at the lab or on the state’s DNA Subcommittee had the expertise to double-check the software, said a scientist in the lab who worked on the techniques who asked to remain anonymous for fear of career repercussions. “We don’t know what’s going on in that black box, and that is a legitimate question,” the scientist said, adding that evidence in older cases should “absolutely” be retested in light of growing questions about FST. “As a scientist, I can’t say no.”
The issue of proprietary "black box" software interpreting DNA mixture evidence mirrors the situation with private vendors used in Texas like STR-Mix and TrueAllele, as discussed on the Reasonably Suspicious podcast.

When New York's proprietary software was picked apart by the defense, they found previously unrevealed errors and flaws. Who's to say that won't also be the case with proprietary private software? The likelihood surely is greater than not.

Courts should allow defense experts access to the source code if they're going to admit DNA mixture evidence from these black box systems. There's just too much recent error and loads of evidence that the math and science involved in these black-box interpretations are well beyond the ken of most analysts seeking to interpret them.

Here are a few other tidbits from the ProPublica/Times story which add new information to the discussion here in Texas:

The FBI in May claimed to have validated STR-Mix for use in DNA mixtures involving up to five people. But having a law enforcement agency "validate" the method is far different from having validation from independent sources which would satisfy defense experts. In one famous case out of upstate New York, STR-Mix and their main competitor, TrueAllele, tested the same evidence and came back with different results.

"In the past three years, flaws in DNA methods have temporarily shut down testing in public crime labs in Austin, Texas, and Washington, D.C."

“Lab analysts 'make it seem like it’s a completely objective process,' said Bicka Barlow, a lawyer in California with a master’s degree in genetics and molecular biology. 'But I’m 100 percent convinced that there are many people who are incarcerated who were convicted with DNA evidence who are innocent.'”


Anonymous said...

“Lab analysts 'make it seem like it’s a completely objective process."

That's because lab analysts want to keep their jobs. Lab analysts are trained to be ignorant and to disregard information that doesn't benefit the prosecution.

Anonymous said...

"...said a scientist in the lab who worked on the techniques who asked to remain anonymous for fear of career repercussions..."

Unfortunately, statements such as this are innate to the forensics field. It's statements such as this that deter honest intelligent scientists from entering into (or staying in) a career in forensics It implies that under anonymity the forensic scientist will give a truthful opinion. But when their name is attached to official statements, such as courtroom testimony or recorded depositions, they will fudge the facts, omit pertinent information, or deny knowledge of problems when they absolutely know of the scientific problems contained within the conclusions. They won't provide the whole truth -- in fear of "career repercussions."

This statement has no place in the scientific field. No scientist should fear losing their job if they expressed a scientific opinion, even if it is unpopular or has tremendous consequences for others. His employers should be removed from authority immediately for requiring the scientist to curtail a scientific opinion.

Anonymous said...

Imagine if more truthful arson investigators from the 1980s-2000s spoke up about the scientifically unfounded myth of "crazed glass caused by liquid accelerants." What if more ballistics experts called b.s. on the certainty of matching a bullet with the gun that fired it. What if more hair comparison experts publicly critiqued the methodology and reliability of accurate "visual expertise interpretation." But these scientists didn't speak out because they feared "career repercussions."

And the public is seeing the consequences of their silence 30 years later.
And probably will for the next 30 years -- because this same attitude persists.

Anonymous said...


To the same...

“Lab analysts 'make it seem like it’s a completely objective process."

Otherwise they'll be fired for making themselves and the crime lab look ignorant. There is no job security, and you could actually be charged with a crime (manipulating evidence, fraudulent reporting, etc) depending on whomever you piss off.

Chris Halkides said...

In a chapter within the new book Forensic Science Reform, Ford and Krane wrote, "When it is necessary to invoke dropout
(and/or drop-in) for test results to be consistent with a prosecution’s theory of a case, it is critically important that both the numerator and the denominator be fully disclosed; yet, some probabilistic approaches currently provide only the ratio of the two likelihoods.Proponents of probabilistic genotyping software packages should scrupulously avoid succumbing to market pressures to have their programs simply "get bigger numbers.'"