Monday, November 30, 2009

Coming to grips with unscientific forensic practices

The Fort Worth Star Telegram yesterday published a lengthy, remarkable piece from Yamil Berard titled "Stakes are high as doubt is cast on forensic lab techniques." The article is one of the first published in the MSM in Texas to fully explore the implications of conclusions published earlier this year by the National Academies of Science (NAS) that many common forensic techniques had no scientific basis and their validity had never been tested - particularly comparative disciplines where individual technicians seek to match patterns in everything from fingerprints to tire tracks to ballistics to bite marks. Reports Berard:
The Texas Forensic Science Commission had planned this month to begin a series of discussions about a national report that opened a Pandora’s box of questions about crime lab techniques. The National Academy of Sciences — advisers to Congress and the president — reported that conclusions about bullet matching are opinion, not fact. Most other identification methods widely used by forensic scientists, the panel advised, also haven’t been validated.
How did a scientifically unproven method receive the blessing of the FBI and forensic "experts" across the nation and other crime lab methods become so widely accepted?
"In a nutshell, these people aren’t scientists," said Jay A. Siegel, a member of the academy, which was established by President Abraham Lincoln to advise the nation on far-reaching questions of science and technology. "They don’t know what validation is. They don’t know what it means to validate a test."
Bullet matching — a practice that takes place every day in Texas crime labs — isn’t reliable, Siegel said, and no studies have been conducted to prove the extent to which firearms marks are unique.
"It’s not possible to state with any scientific certainty that this bullet came from any weapon in the world," said Siegel, who is the chairman of the department of chemistry and chemical biology at Indiana University-Purdue University Indianapolis and director of the Forensic and Investigative Sciences Program.
Other identification methods — including tying hairs to suspects, guns to criminals and blood spatter to crime scenes — lack protocols and standards that legitimize such practices as "scientific."
Of the methods, only DNA analysis stands up to the test to match an individual to crime evidence, the scientists reported.
One of the Forensic Science Commissioners Gov. Perry recently declined to reappoint told the Startlegram that ""The NAS opened the [Pandora’s] box, and now we just need to face it." Everyone interested should read the whole piece.

Berard predicts that, if the Forensic Science Commission restarts its plans to sponsor discussions around the state about the implications of the NAS report, "criminalists and crime lab directors across Texas will likely blast the academy’s report, 'Strengthening Forensic Science in the U.S.: A Path Forward.'" I've not heard anyone "blast" the findings per se, but maybe that's coming. (A discussion of the NAS report appears on the first day's agenda of an upcoming conference in Fort Worth on "Current Trends in Forensic Science," so it's possible that might be a place where reservations are more frankly expressed.)
Relatedly, the Waco Tribune-Herald had a story yesterday on the current state of arson science and the impact of critiques of outdated forensics in the Todd Willingham case on the field of arson investigation ("Local fire investigators confident in arson training techniques and welcome scrutiny," Nov. 29) According to reporter Cindy Culp:
Critics of Texas’ fire investigation standards worry that some investigators may still cling to the old wives’ tales, however. They also express concern about investigators not being required to have a science background, saying it is needed to truly understand fire behavior.
In Texas, people can become fire investigators as long as they have a high school diploma, complete 150 hours of training and pass a certification exam. To investigate arsons, they also have to become certified as a peace officer, which generally involves going through a police academy.
Central Texas officials who conduct fire investigations say there may be truth to some of the criticism. But, by and large, investigators do a good job, they say. ...
[Killeen Fire Marshal James] Chism said the controversy over the Willingham case is a good chance for the profession to look at old investigations where questionable techniques may have been used. The industry also needs to seize the opportunity to weed out investigators who cling to outdated beliefs. It would be naive to think none exist, he said. “Whether it is an old-school mentality or sheer laziness because it’s what they’ve always done, I still have to think those old wives’ tales are still getting play in the state of Texas,” Chism said.
In the end, Chism predicted, public confidence in fire investigations will increase.
Over the next five to ten years we're going to see an historic, first-of-its-kind evaluation of so-called forensic "science" disciplines that are really subjective, untested and in some cases inherently unscientific.
That's the real concern with John Bradley shutting down the Forensic Science Commission debate over the Todd Willingham case. As the Killeen fire marshal says, it would be naive to believe other investigators aren't still coming up with arson findings based on assumptions that aren't supported by science. But even more importantly, there are many other forensic disciplines that we now know are equally deserving of open, honest scrutiny. Unless Mr. Bradley does an unexpectedly bang-up job that assuages widespread concerns that he's engaged in a coverup, the main entity created by the state to spearhead such evaluations won't have the public confidence required to be seen as a neutral arbiter.

Related Grits posts:


Anonymous said...

DNA might not even be so reliable, according to Israeli scientists...

"Israeli Scientists Prove DNA Evidence Can Be Faked"

Anonymous said...

It was the bread mold that did it. It is amazing that something so simple as bread mold could change the way a society is organized and lead to incredibly large scale social policies that are so terribly misdirected.

Scientists, particularly of the sort that we normally think of in forensics, became respectable after the medical profession discovered that bread mold could cure people who normally died. Yes, it was penicillin. You see, before penicillin, if you got sick, you either got better or you died. There were doctors, but they didn't have much impact. And because they didn't have much impact on life and death, they didn't get much respect. If you wanted to prove something back before doctors became gods, you didn't resort to microscopes and chemicals and weird looking instruments. Those were just things that these strange men in lab coats played with; but, like astrologers, when it came down to it, they didn't have much to offer. The result was, people didn't listen to them very much, and they didn't believe much of what they had to say. If you wanted to prove something back then, you resorted to logic about the things that were obvious for everyone to see, and then you went to the Bible. This held a whole hell of a lot more water than some wanker with test tubes, a bunch of stained papers, and an incomprehensible theory about things people couldn't see.

But when people saw their relatives get well in a few days after having the same sickness that killed another relative a few years earlier, they started thinking that maybe these rascals in weird clothes who played with chemicals and strange looking tools might be onto something. Over a period two or three decades after they started curing infection, the new arbiters of "truth" in the world were "scientists". The doctor suddenly knew something that was more important than what the Bible said or what you could deduce with your own logic from the things that you could see. Relatively rapidly, we gave up on the old ways of knowing truth, and put all our faith in anyone who looked like a scientist. If they told us to jump, we asked how high. If they told us we were going to die, we wrote our will.

By the mid 1940s, the best way to convince anyone in the United States of anything was to have a doctor or a scientist tell them it was so. People believed anything they said, and they believed it unconditionally, no matter how much it conflicted with appearances or common sense. These people -- the ones who could pass as "scientists" -- suddenly had the unconditional power that priests, shaman, medicine men, kings, and other leaders had sought and devised ways of getting throughout all of recorded history and even before. They had the power to make people believe anything, and they had the power to make people believe the opposite of what their common sense suggested.

It doesn't take long for power like that to become sought by every person with a vested interest in what others believe. Enter the forensic science expert witness. The prosecutor's dream! The person who the jury would believe even if every other available shred of evidence contradicted them. If they had a white coat, by god, whatever they said was "a fact".

That's how we got here. That's how it all got started. Knowing how it all got started can sometimes help us see where we went wrong, and what we might do about it.

Chris Halkides said...


Radley Balko has a good interview in the Economist in which he discusses forensics ( He mentions the following article: