A forensic science 'blockbuster' and the limits of accreditation

A couple of forensic science related items before I turn my attention away from the blog for the day:

NAS Forensic Science Report a 'Blockbuster'
In an interview, White House science adviser John Holdren was asked, "What single published finding or scientific paper has had the biggest impact on your office?" He gave several examples, but notably mentioned:

[Another is] the Academies' report that came out last year on forensic science that suggested much of forensic science is less solid than is assumed. We ended up spending a lot of effort, and we're still spending a lot of effort on what needs to be done to shore that up and what are the policy implications. In the meantime, we're working with the Justice Department. That [report] was quite a surprise. I'd not given a lot of thought to the scientific foundations of forensic science. But that one was a blockbuster.

"Blockbuster" is the right word for it. I'm still startled when folks in criminal justice fields who work routinely with forensic examiners aren't conversant about the NAS report on forensics or pooh-pooh its importance. Others take the NAS report as a personal affront, but at least that unnecessarily defensive reaction denotes a recognition of what a big deal it is. Like it or not, that report re-set the terms of debate over forensic errors for probably the next couple of decades, which is how long, at a minimum, I expect it will take to perform the necessary scientific research to validate or refute longstanding forensic methods from ballistics to fingerprints, many of which simply have no scientific basis.

Thoughts on Accreditation and Forensic Reports
At the Court of Criminal Appeals' Forensic Science Seminar, I attended a presentation by Ralph Keaton from ASCLD-LAB (see his power point presentation here), which is the largest accreditation agency for forensic sciences. Accreditation has come a long way; Keaton said that 25 years ago, not a single lab in the country, including the FBI's, would meet their standards today. Even so, until 10 years ago ASCLD-LAB was run out of somebody's house, and they had no staff before 1995. Today they employ 13 people to overview crime labs across the country. Their prominence has been boosted by states like Texas, which began requiring accreditation in 2003 for certain types of forensic evidence to be submitted in court. The state has accredited 90 crime labs (not all by ASCLD-LAB) to submit evidence in Texas courts; half are outside the state.

Even so, lab accreditation isn't a cure-all: Once they're accredited, ASCLD-LAB only requires renewal once every five years, when labs submit a "conformance file" for assessment, and shorter annual reports in between. What's more, the labs themselves choose what cases inspectors examine, so there's a risk they clean them up ahead of time or cherrypick examples. Keaton insisted that if that was occurring, they didn't do a good job of covering their tracks because they "always" found something.They also require proficiency testing of lab workers and do interim inspections related to allegations.

Anyway, readers will recall that it came out at that seminar that DPS did not disclose disagreements among fingerprint examiners in their final reports. I was later told that was the agency's practice not just in fingerprints but in other comparative disciplines like hair, ballistics, tool marks, etc.. Under the ACEV method (Analysis, Comparison, Evaluation and Verification), when an examiner concludes that two items match, a second examiner is brought in to verify the finding. When those examiners disagreed, the case went to a supervisor for resolution. In situations where the supervisor decided in favor of the examiner who thought a match was found, the disagreement was not being reported even to prosecutors, much less to the defense, in the crime labs' final reports.

I'd noted before that the failure to be forthcoming about in-lab disagreements among examiners violates the state's obligation under Brady v. Maryland to disclose exculpatory evidence to the defense. But looking again at my notes from the presentation on accreditation, it also strikes me that the practice probably violates the ASCLD-LAB accreditation standard passed in 2005 that requires issuance of reports for all work performed. I was recently told DPS is in the process of "fixing" that oversight - I need to check on the status of any changes in that regard - but even if they do, there are an unknown number of past cases where disagreements weren't disclosed, and every reason to believe that other, non-DPS labs across the state are doing the same thing. What should be done about that?

Unfortunately, accreditation doesn't solve the problems raised in the NAS report because labs are accredited based on compliance with existing practices that simply haven't been validated by science one way or the other. But if its strictures were meaningfully enforced, accreditation might help prevent structural errors like the failure to disclose examiner disagreements by DPS.