Sunday, December 26, 2010

A forensic science 'blockbuster' and the limits of accreditation

A couple of forensic science related items before I turn my attention away from the blog for the day:

NAS Forensic Science Report a 'Blockbuster'
In an interview, White House science adviser John Holdren was asked, "What single published finding or scientific paper has had the biggest impact on your office?" He gave several examples, but notably mentioned:
[Another is] the Academies' report that came out last year on forensic science that suggested much of forensic science is less solid than is assumed. We ended up spending a lot of effort, and we're still spending a lot of effort on what needs to be done to shore that up and what are the policy implications. In the meantime, we're working with the Justice Department. That [report] was quite a surprise. I'd not given a lot of thought to the scientific foundations of forensic science. But that one was a blockbuster.
"Blockbuster" is the right word for it. I'm still startled when folks in criminal justice fields who work routinely with forensic examiners aren't conversant about the NAS report on forensics or pooh-pooh its importance. Others take the NAS report as a personal affront, but at least that unnecessarily defensive reaction denotes a recognition of what a big deal it is. Like it or not, that report re-set the terms of debate over forensic errors for probably the next couple of decades, which is how long, at a minimum, I expect it will take to perform the necessary scientific research to validate or refute longstanding forensic methods from ballistics to fingerprints, many of which simply have no scientific basis.

Thoughts on Accreditation and Forensic Reports
At the Court of Criminal Appeals' Forensic Science Seminar, I attended a presentation by Ralph Keaton from ASCLD-LAB (see his power point presentation here), which is the largest accreditation agency for forensic sciences. Accreditation has come a long way; Keaton said that 25 years ago, not a single lab in the country, including the FBI's, would meet their standards today. Even so, until 10 years ago ASCLD-LAB was run out of somebody's house, and they had no staff before 1995. Today they employ 13 people to overview crime labs across the country. Their prominence has been boosted by states like Texas, which began requiring accreditation in 2003 for certain types of forensic evidence to be submitted in court. The state has accredited 90 crime labs (not all by ASCLD-LAB) to submit evidence in Texas courts; half are outside the state.

Even so, lab accreditation isn't a cure-all: Once they're accredited, ASCLD-LAB only requires renewal once every five years, when labs submit a "conformance file" for assessment, and shorter annual reports in between. What's more, the labs themselves choose what cases inspectors examine, so there's a risk they clean them up ahead of time or cherrypick examples. Keaton insisted that if that was occurring, they didn't do a good job of covering their tracks because they "always" found something.They also require proficiency testing of lab workers and do interim inspections related to allegations.

Anyway, readers will recall that it came out at that seminar that DPS did not disclose disagreements among fingerprint examiners in their final reports. I was later told that was the agency's practice not just in fingerprints but in other comparative disciplines like hair, ballistics, tool marks, etc.. Under the ACEV method (Analysis, Comparison, Evaluation and Verification), when an examiner concludes that two items match, a second examiner is brought in to verify the finding. When those examiners disagreed, the case went to a supervisor for resolution. In situations where the supervisor decided in favor of the examiner who thought a match was found, the disagreement was not being reported even to prosecutors, much less to the defense, in the crime labs' final reports.

I'd noted before that the failure to be forthcoming about in-lab disagreements among examiners violates the state's obligation under Brady v. Maryland to disclose exculpatory evidence to the defense. But looking again at my notes from the presentation on accreditation, it also strikes me that the practice probably violates the ASCLD-LAB accreditation standard passed in 2005 that requires issuance of reports for all work performed. I was recently told DPS is in the process of "fixing" that oversight - I need to check on the status of any changes in that regard - but even if they do, there are an unknown number of past cases where disagreements weren't disclosed, and every reason to believe that other, non-DPS labs across the state are doing the same thing. What should be done about that?

Unfortunately, accreditation doesn't solve the problems raised in the NAS report because labs are accredited based on compliance with existing practices that simply haven't been validated by science one way or the other. But if its strictures were meaningfully enforced, accreditation might help prevent structural errors like the failure to disclose examiner disagreements by DPS.


Prison Doc said...

As bad as I hate to agree with John Holdren on anything, reports such as the one cited have sure been an eye opener to me. As a scientist by training I've long been a fan of DNA evidence, but the blockbuster to me has been the questioning of fingerprint evidence, since we've all been trained since toddlerhood by TV crime shows that fingerprints never lie.

Anonymous said...

Accreditation is a joke. Crime labs pay a small fee for the certificate and then wildly wave it around anytime an accusation of misconduct arises.

And most of the time, the scientific misconduct "found" by ASCLD-LAB or auditors had been occurring for many, many years (and thus, should have been "found" many, many years earlier, if not by auditors then by the crime lab itself.)

Anonymous said...

This might be a bit late, but I just found the 3-page summary of the APD audit findings (ex-analyst Hamilton allegations)

There seems to be quite a few subjective words such as "adequate", "appropriate", and "sufficient". Are these words that should be used in the forensic sciences?

And the APD protocols for Corrective Action Reports appear to be backwards in that DNA contamination found after evidence is submitted to the DNA lab is written into a memo placed into the case file (thus appearing as a one-time-only event) while DNA contamination found before submission to the DNA lab is documented in a Corrective Action Report. Thus, systemic contamination problems inside the DNA lab would only be detected by (luckily) finding memos in several different case files and making the connection. This also means that information related to systemic problems are inaccessible to the public because the memos are case-specific, privileged information. Corrective Action Reports, OTOH, are subject to public availability because of the Public Information Act, but generally not picked-up by prosecutors and criminal defense attorneys because they are only concerned with case-specific info.

For complete transparency, the Corrective Action Reports need to document the DNA contamination events that are discovered both before AND after submission of evidence to the DNA lab.

The summary also mentions "National Standards". Anyone know what these are?

And "Yes", the authors did mention the "sufficient" reviews in the external audits of the previous 5 years.

Maybe Bradley and the Forensic Science Commission can decipher this summary better...