The Austin Police Department has temporarily suspended operations at its DNA lab because of concerns raised by the Texas Forensic Science Commission, a lack of properly trained supervision and the need to allow staff to learn a new federally required way of verifying evidence, the American-Statesman and KVUE-TV have learned.
The decision means that hundreds of DNA samples — often crucial evidence in crimes such as homicides and sexual assaults — will either be shipped to the Texas Department of Public Safety lab or to private labs for analysis, possibly delaying the outcome of pending cases in already backlogged Travis County criminal courts.The article declared that the closure "comes at a time when DNA labs nationally are adapting to new federally required procedures." However, when I emailed the FSC's Lynn Garcia, she said she's "not sure what federal guidelines they are referring to." Really APD just screwed up the science, using a method which is "neither scientifically valid nor supported by the forensic DNA community," according to the six-page letter from the FSC to Austin PD critiquing their methods, which Garcia provided to your correspondent.
Embarrassingly, according to the letter, the main problem was that lab personnel didn't understand past federal directives: "Technical Leaders (TLs) and senior analysts in the APD DNA Lab appear to have misunderstood language from the Scientific Working Group on DNA Analysis Methods (SWGDAM) Interpretation Guidelines for Autosomal STR Typing by Forensic DNA Testing Laboratories ... and from Dr. John M. Butler's textbook, 'Advanced Topics in Foensic DNA Typing: Methodology.'"
"Of greater concern, the analysts themselves were aware the [method] was ineffective because they observed ... [problems] in their own casework" and did nothing. They cherrypicked data, choosing which loci to compare based on whichever ones show up in the sample. "The appropriate approach is to decide which locus (or lici) should be used first ... as indicated by the overall analysis of the evidentiary sample, not on which alleles are present or absent based on the victim or suspect known profiles."
The FSC identified at least one case where contamination made APD's results invalid, as confirmed later by a private lab. They also found ten cases of contaminated reagents.
Worryingly, the problems were not revealed by ASCLD/LAB assessments, even though deficiencies were evident in the data they were given.
These observations raise legitimate questions regarding the limits of accreditation and the consistency of assessor teams. Specifically,: (a) Are the scope and limitations of accreditation well understood by the criminal justice community? (b) Do assessors consistently consider whether the laboratory's protocols and underlying validation based on sound scientific principles or do they limit their review solely to determining whether the laboratory has a protocol in place that it follows? (c) Should assessors re-review validation data from prior years considering that validation studies are relied upon to build subsequent protocols?Most of that detail wasn't included in the Statesman story.
Concluded the paper, "the Austin Police Department’s crime lab, which will have to recalculate statistics on about half of the 1,297 Travis County cases identified so far, is still validating new software and updating its protocols. Meanwhile, the lab’s backlog of cases awaiting DNA analysis has risen to about 1,300, the most in the past five years."
It's a mess, but it's not that different than what most other crime labs in the state are going through, except most of them didn't shut down operations until their staff are retrained. After all, the errors relate only to mixtures. There are still quite a few cases, including most rape kits, where there is only one unknown sample. That part Grits finds odd, but I'm glad they're belatedly getting their act together on this topic.
MORE: From Forensic magazine.
See prior, related Grits posts:
- A reluctant scoop: Changing interpretations of DNA mixtures vex legal system
- Labs must correct wrong DNA mixture analyses, learn when to recognize 'crap'
- Resources on DNA mixtures
- The challenge of notifying defendants in large-scale forensic error cases
- DNA mixture SNAFU a mess but don't expect 'deluge' of innocence claims
GFB,
ReplyDeleteI have never worked for the APD, and have never audited their DNA program, so my comments are based solely on my reading of the TFSC letter, the various news reports, and my experience in science as a university and medical school researcher and faculty member, and in forensic DNA science in particular.
First, I would take exception to a couple of your assessments.
1. "After all, the errors relate only to mixtures." The issue with stochastic thresholds does not relate only to mixtures. It also relates to low-level and partial single source samples where the level of signal is so low that the analyst can not have confidence that all of the genetic information in the sample is being detected. Without a correct, statistically based stochastic threshold, there is no way to calculate a correct match statistic for a low-level or partial DNA profile. The match statistic will either be overstated because the analyst interpreted some loci as homozygotes when there was insufficient justification, or understated because the analyst incorrectly used a dropout calculation at some loci when there was sufficient signal to confidently eliminate the possibility of dropout.
2. "It's a mess, but it's not that different than what most other crime labs in the state are going through..." As the TFSC letter states, following the 2010 SWGDAM guidance document, APD was the only laboratory in Texas that implemented this incorrect approach to setting a stochastic threshold. All other labs that implemented stochastic thresholds after 2010 implemented a correct approach, and some labs had implemented the correct approach long before the 2010 SWGDAM document. The APD approach was an obviously incorrect approach. Had the APD asked any of the medical school-affiliated forensic labs in the state (there are several) for feedback, they would have known that their approach was woefully incorrect.
In my view, there are two fundamental root causes at play here, neither of which is addressed by the APD's stated correction plan.
First, forensic DNA science has become very statistically based. The level of statistical understanding to properly develop and implement procedures is beyond the level provided by masters-level training programs, which is the degree that most DNA technical leaders have. The APD should be hiring a PhD level scientist as technical leader.
Second, this fiasco is a predictable result of a police department trying to run a high end science program. Police departments are poorly suited to do good science. The culture and management structure of police departments are resistant to change, and that cultural resistance to change can be seen here in the failure of APD to do anything in response to the fact that "the analysts themselves were aware the [method] was ineffective because they observed ... [problems] in their own casework". APD should stop trying to do science.
Thanks for your comments 9:55, that's helpful.
ReplyDeleteIn response, I'll defer on your first point - I thought the stochastic threshold issue related mainly to mixtures, but if it also applies to other low-signal samples, so be it. I'm absolutely unqualified to have an opinion on the science, I'm just a layman trying to make sense of this from what's being said publicly by experts.
On the second point, it's not my understanding that "All other labs that implemented stochastic thresholds after 2010 implemented a correct approach." For example, experts presenting at this FSC forum last year didn't think so. I agree APD was an outlier, using a method nobody else used, but my understanding was that plenty of others had to go back, change their procedures, and retrain as well.
Totally agree with your last two points. The math on this is head-spinning stuff and the cops-doing-science problem is especially poignant when the math/science is so complex.
So, forensic analyst Cecily Hamilton was correct in her complaint to the TFSC in 2010. In 2010 APD crime lab DNA analyt Cecily Hamilton alleged that quality assurance personnel at the lab were underqualified and she knew of two separate cases of "contamination" at the lab, and that staff were threatened with retaliation if they reported problems in the lab.
ReplyDeleteInexplicably, a joint review by the Texas Department of Public Safety and the Texas Rangers had cleared the Austin Police Department's crime lab of allegations made by the former employee in 2010.
Though an investigation said claims were unfounded, at the time Cecily Hamilton said in a public statement that "APD is covering up the fact that their DNA lab has issues and that they performed a bogus internal investigation and they are trying to discredit and slander me so that people will not listen when I tell the truth about what occurred during my employment there."
The TFSC ate the shit sandwich they were fed, and ignored the analyst's concern.
Business as usual.
Proof:
ReplyDeletehttps://sliterchewspens.files.wordpress.com/2013/03/slide93.jpg
https://sliterchewspens.files.wordpress.com/2013/03/slide94.jpg
https://sliterchewspens.files.wordpress.com/2013/03/slide100.jpg
https://sliterchewspens.files.wordpress.com/2013/03/slide103.jpg
https://sliterchewspens.files.wordpress.com/2013/03/slide109.jpg
https://sliterchewspens.files.wordpress.com/2013/03/slide110.jpg
-SCP
Had forgotten about Cecily Hamilton but you're right, 11:12. Chief Acevedo held a press conference back in 2010 aiming to discredit her but she looks awfully prescient right about now.
ReplyDeleteThat said, to be fair, the Rangers and DPS may not have been the right reviewers to catch such errors (DPS was doing this wrong, too, if wrong, differently, until last year). As the commenter at 9:55 opined, they probably need a PhD level scientist/mathematician to stay abreast of the nuances surrounding high-end DNA testing, if they want to continue doing it. Some of the other disciplines, like toxicology, are a little more plug-and-play. But these analyses require subjective interpretation and as near as I can tell, there are quite a few analysts out there using methodologies which they do not truly understand.
This exact compliant was presented to the TFSC in 2009 regarding the Dallas County Crime Lab's use of expired and incorrect chemicals (SWIFS) for preparing reagents. At least a half-dozen chemicals in the Serology lab were being used well beyond the expiration date -- one of the chemicals was as old as 14 years past expiration. No validation testing was performed using the expired chemicals to empirically determine the efficiency of the chemicals. Once the complaint was submitted, and their stupidity revealed, the SWIFS Lab Management (Tim Sliter and Stacy McDonald) threw away the expired chemicals so that they could not be evaluated for error rates or effectiveness. Although there is no documentation, lab analysts at the time could tell you that these reagents (when prepared with the expired chemicals) had about a 30% error rate.
ReplyDeleteThe TxDPS did nothing.
ASCLD-LAB wrote a bogus and fabricated report (never visited the lab for an on-site audit), providing an unscientific and imbecilic explanation.
The TFSC whitewashed the investigation, and failed to acknowledge the existence of the complaint.
SWIFS Supervisor Stacy McDonald testified in court that the "analysts were negligent" (not her). SWIFS did not "Self-Disclose" this malfeasance to the TFSC.
This complaint is not listed on the TFSC webpage.
-SCP
"...the Rangers and DPS may not have been the right reviewers to catch such errors..."
ReplyDeleteThen why were they asked to investigate, and why did the TFSC view their reports as credible (writing their own report too)?
It's either corruption for cover-up, or Texas is by far the most forensically ignorant state in the Nation. Dip-shittery abound!
Cecily Hamilton has a very strong civil case for wrongful termination and slander/libel against Austin Police Chief Art Acevedo, Texas Rangers, ASCLD-LAB, TFSC, etc.
I suppose, 12:52, the answer is that they did not at the time enjoy the benefit of 20/20 hindsight, as we do here.
ReplyDeleteYes, in retrospect it's clear Cecily Hamilton was right and Acevedo's attacks on her look a lot more today like a cover up. Granted. If her civil cases is still going, I hope this helps it. The FSC I'm less inclined to accuse of a cover up. They could only base their judgments on the information and representations Austin PD gave them. The thing about cover ups is, they tend to conceal things.
We really need this high quality journalism in other states.
ReplyDeleteI might add, that simply being a PhD is no cure for basic academic understanding of population statistics and statistical modeling. The Federal requirement is at least a Masters degree for a Technical Leader. You need someone who is both academically engaged but also who can get the job done and not lose the forest for the trees. I have known many a PhD who get so caught up in the academics that they grind to a halt and are unable to see a project (or validation) to its successful completion. In the end, you need a successful procedure to get the forensic cases both in and out the door.
ReplyDeleteScary New FBI rapid profile DNA Bill
ReplyDeletehttp://www.nextgov.com/defense/2016/07/house-committee-approves-bill-letting-fbi-use-rapid-dna-profiling/129735/