Saturday, October 09, 2010

Brady violations by DPS fingerprint examiners? Is fingerprint examination even science?

There was an astonishing moment yesterday at a breakout session on fingerprint examination at the Texas Forensic Science Seminar, at which Department of Public Safety fingerprint examiner Bryan Strong (who seemed like a really nice guy so I hate to pick on him) was describing how his division implemented the ACEV method of fingerprint examination in ways that may violate the state and prosecutors' obligations under Brady v. Maryland.

ACEV stands for Analysis, Comparison, Evaluation and Verification. That's bureaucrat-speak for looking at the fingerprints visually and subjectively deciding if they're the same based on "training and experience" (as opposed to any sort of objective standard), then having a second examiner look at them to "verify" the results. There is no minimum number of similarities or comparison points required to declare two fingerprints a "match," though many other countries have established such standards. (Notably, at DPS if an examiner finds fewer than 11 points of comparison, two people must verify the conclusion.)

Anyway, Mr. Strong described what happens when the first examiner finds a match but the verifying analyst doesn't agree. In such instances, he said, they notified their supervisor and all of them conferred to make a decision. A defense attorney in the crowd asked what seemed to me an obvious question: When two examiners originally disagreed but a supervisor resolved the issue in favor of a match, was that disagreement recorded in the final report? No, replied Strong, only the conclusion. At this, the audience began to murmur and fidget. Somebody from the back cried out, "Have you ever heard of Brady v. Maryland?," which is the US Supreme Court case requiring the state to turn over all exculpatory evidence to the defense  before trial. No he had not, replied a credulous Strong, a statement which elicited an audible gasp from the crowd.

So essentially, if two examiners who looked at the prints come to different conclusions but a supervisor resolves the question against the interests of the defendant, according to this presentation, that information is not routinely disclosed to defense counsel. On its face that's a straightforward violation of Brady. Who knows how many times that scenario has occurred over the years!

A representative from the Texas Attorney General's office then asked Mr. Strong if his division had access to legal counsel at DPS, and he said he believed they did. She told him politely (if somewhat obliquely) that it appeared there were some legal issues surrounding the division's work that he wasn't aware of (problems that likely emanated higher up the chain of command than Strong's level, she gently added) and offered her agency's assistance to retrain folks at the fingerprint examination division on the subject!

There was quite a bit of discussion of fingerprints at the event, some of the most interesting by Dr. Jay Siegel, a forensic scientists who was on the 17-member National Academy of Sciences panel which published a damning report last year calling into question the scientific basis of "pattern evidence," where visual comparisons were the basis for connecting evidence to a defendant. Fingerprints are by far and away the most common and important type of pattern evidence. Siegel says 60% of fingerprint analysts don't actually work in crime labs - they're sworn police officers working in their own departments.

The most high-profile case of a mismatch was that of Brandon Mayfield, an Oregon attorney falsely accused of the Madrid train bombing after four examiners mistakently identified him using the ACEV method. (See the USDOJ Office of Inspector General's report [pdf] on that case.)

Siegel cited eye-popping data from a study published in the 1990s in which the umbrella organization for fingerprint examiners, the  International Association of Identification, performed proficiency testing on their own members and came up with a 19% false-positive rate! Though I can't find the study online, I did find this apparent reference to it in an article from the Los Angeles Times:
In a 1995 IAI-approved proficiency test, 22% of the test takers identified the wrong person one or more times. In addition, 36% could not identify prints that test givers said they should have been able to match, which means that a guilty person could have gone free in a real case.

Ken Smith, chairman of IAI certification, suggests the error rate may not fairly represent the profession because the test takers were anonymous and there was no way to determine their credentials.

But fingerprint examiner David Grieve, editor of the Journal of Forensic Science, said in an article that the test results were alarming and noted that reaction in the forensic science community "ranged from shock to disbelief."
That's just one study, Siegel emphasized, and more research is needed in the area, but it's certainly a stunning result. The day before Dr. Joe Bono, President of the American Academy of Forensic Sciences, had said a technique which had a 30-40% error rate (the Horizontal Gaze Nystagmus test for intoxication used in DWI enforcement) didn't qualify as "science." I asked him if a technique resulting in a 19% false positive rate constituted science, but he demurred, saying he hadn't seen the fingerprint study. Siegel said proficiency testing of all forensic examiners, including those dealing with fingerprints, should be "blind," i.e., given to the examiner as part of their routine work so they didn't know they were being tested.

A good argument for the "blind" testing approach may be found in the work of cognitive neuroscientist Itiel Dror, which Dr. Siegel cited and which has previously been discussed on this blog,  who conducted a study in which five fingerprint examiners were given pairs of prints which they'd earlier personally matched during their own career. This time, however, they were told the prints were from the Brandon Mayfield case, which had already been well-publicized. The result: Three of them reversed their conclusion, saying the prints they'd previously "matched" did not come from the same person. A fourth said the results were inconclusive. Only one of the five stuck to his guns and said the prints came from the same person!

This makes me wonder if the ACEV approach itself is fundamentally flawed by bias. If knowledge of another examiner's conclusions can so easily taint results, it shouldn't be the case that the second analyst should know that they're being called on to "verify" someone else's match, which implies someone else already reached a conclusion. It seems to me it'd be much cleaner to give the prints to the second examiner without telling them what the first examiner found.

Siegel concluded that there's no scientific proof fingerprint examination (or for that matter, other pattern evidence) can "individualize" their results to the exclusion of other possible sources. Indeed, searching around this morning for the IAI study, I ran across these comments from Dr. Siegel in another forum last year where he persuasively argued that individualizaiton isn't even necessary:
Not only do I beleive that a conclusion of absolute inclusion is not scientifically justified but that it is not necessary. I don’t believe that the concept of “individualization” has any place in science and is not provable in the real world. Why is it necessary to offer a nonsupportable, unprovable conclusion in court, where there is so much at stake and juries are so easily mislead? When a forensic scientists analyzes evidence that is brought to the lab by a criminal investigator, there is a reason why the focus is on that person. (This can set up a situation that invites bias on the part of the examiner, but that is another issue). The fact that there are many similarities between a latent print lifted at a crime scene and a print from a suspect, with no unexplainable differences, should be testimony enough. This would presumably be one piece of corroborating evidence in a net of evidence being offered by the prosecutor. There is no need to “guild the lily” by adding the conclusion that there is no other fingeprint in the world that could be the source for the print from the crime scene. Offering unsupportable conclusions in court reinforces the idea that forensic science really isn’t science; it is a tool of the prosecution.
That position makes a lot of sense to me. The conclusion of a fingerprint "match" by an "expert" is an incredibly damning piece of evidence when really all they're saying - that anyone can prove - is that there are similarities between the two prints. There were also similarities between Brandon Mayfield's print and the Madrid train bomber's, but that didn't mean Mayfield did it.

18 comments:

  1. Another type of pattern evidence is microscopic sperm cell identification (from sexual assault evidence). The crime lab I know uses the “Christmas Tree” staining procedure where the sperm bodies become stained pinkish and the tails become stained green. The problem is...most sperm cells easily loose their tails over time and just about everything else on the microscope slide also becomes pink-stained (other cell-types and debris or whatever). So identifying a pink oval object on a microscope slide becomes extraordinarily subjective. If two analysts don’t agree that a particular pink blob on a slide is a sperm cell, a supervisor is supposed to make a decision. And if the supervisor says “yes it is”, the results are recorded as “positive” and the dissenting opinion is not recorded. A single sperm cell might be enough evidence to get a guilty plea to sexual assault. Sounds like another case of a Brady Violation.

    ReplyDelete
  2. It seems some Measurement Systems Analysis tools such as commonly used in industry are called for.

    This just blows me away. When someone in these crime labs is feeling a little sick do they bleed 'em with leeches?

    ReplyDelete
  3. Since you seem interested, how might I forward to you information about the 1995 IAI study?

    ReplyDelete
  4. "A representative from the Texas Attorney General's office then asked Mr. Strong if his division had access to legal counsel at DPS, and he said he believed they did. She told him politely (if somewhat obliquely) that it appeared there were some legal issues surrounding the division's work that he wasn't aware of (problems that likely emanated higher up the chain of command than Strong's level, she gently added) and offered her agency's assistance to retrain folks at the fingerprint examination division on the subject!"

    So to me... this sounds like a discovery/admittance of sorts that the DPS fingerprint exam division has not been following a legal precedent as established by the Brady case(forgive me if I am using the wrong words, hopefully you can figure out what I am saying).

    My question is, now that it has come to light that DPS was not doing what it should legally have done, what kind of can of worms might this open?

    Will the TX AG office just retrain DPS employees, and an attitude of no harm, no foul be taken? Or will this open up the possibility of some kind of appeal being filed by a person who was convicted based on fingerprint evidence? Thus leading to more appeals based on this technicality?

    Long story short... what are the possible ramifications of these "legal issues surrounding the division's work" now that it has been brought to light?

    ReplyDelete
  5. Yes, we must discredit fingerprint evidence as well.

    The criminal defense attorney’s goal when formulating strategy is to discredit the prosecution’s evidence and obtain a “not guilty” verdict at the trial. Everything must be discredited. No matter what, the goal is to have the defendant walk out the door a free man.

    ReplyDelete
  6. Thomas, send it to shenson@austin.rr.com.

    11:40 - Re-read the final quote from Siegel and my conclusion. I don't think fingerprint evidence should be kept out of court, but subjective comparisons shouldn't be confused with "science" and they shouldn't be allowed to overstate their conclusions or hide conflicting views among analysts from the defense.

    ReplyDelete
  7. 11:40 You have heard of the concept of garbage in, garbage out? Or to quote "For making correct decisions, relying on a measurement system and the same for the statistical method used, is necessary as it could be said, when the data quality are low the
    benefits of a measurement system is also low; likewise when the data quality is high, the benefit is high too."

    Do you believe the state (We the people) have an interest in treating fingerprint examination as a revealed religion?

    This is are you smarter than a fifth grader stuff. You propose a system in which in our state crime labs crimes are committed. For what interest does this serve? Certainly not the Due Process clause.

    ReplyDelete
  8. "information about the 1995 IAI study?"

    Will you please post the link on GRITS for the rest of us that may want to read it? Thanks.

    ReplyDelete
  9. "No matter what, the goal is to have the defendant walk out the door a free man."

    No, the goal is for the trial to be fair and for the prosecution to prove its case. If I were to take you ignorant attitude and apply it to the other side I might say: "the goal is to convict the defendant, guilt or innocence doesn't matter."

    How about we just do away with those pesky defense attorneys. Then we wouldn't need trials. You get arrested you go straight to prison. Its because of those defense attorneys that we have 4th amendment rights. The courts have set limits on the things police can and cannot do in cases wheere, the issues were pressed by those sleazy defense attorneys. If it weren't for them, there would be nothing to stop the police from stopping people at random on the highways, searching their cars with no probable cause, taking you to the station for a little "questioning", or from coming into your home and searching just because they feel like it, etc. etc. And, here's the kicker, in most of those cases where courts set those limits, the defendant was actually guilty. But, there's a basic concept that seems simple but so many people seem to lack the insight to grasp. If we don't protect the other guy's rights, even the guilty, we lose our own. You need to be thanking those defense attorneys. While the 4th amendment is pretty weak at this point, the only reason it has any life at all, and the only reason you enjoy any of those rights and protections is because of defense attorneys.

    ReplyDelete
  10. To clarify from being at the conference and having some idea of what the procedures are in fingerprint labs, etc., I don't think DPS was intentionally concealing the information. The problem is a misunderstanding between the lab, the prosecution, the defense bar, and the courts, which is the case with many labs.

    These labs don't hide or leave out the information. They document the disagreement or non-verification in the file. They just don't put it in the report. The problem is that most of these lab employees have been under the belief that defense counsel had access to the file, that the prosecution had access to or read the file, that everybody was on the same page and knew what was going on. They have been wrong, but that doesn't show any bad faith.

    The goal of conferences like these with all parts of the system involved is so that people from each area can communicate. It appears that something good may come of this one-- labs are now aware, or at least this one is, of the steps they need to take and procedures they should put in to place.

    It isn't the place of Bryan Strong to know legal precedent and this was not his fault. This was a breakdown in communication that became clear in a very public way, a lack of communication from all parties. As for the offer from the AG for legal help, my question is why they weren't there before. It is there job, as attorneys for the state, to know this legal precedent and make sure it is followed. The fault goes to the attorneys for the state, not to the lab techs. Blaming the crime lab for not knowing the legal precedent when they are non-lawyers is the same as blaming a judge for not knowing all the medical facts needed in an opertaing room for a malpractice case. The judge has to rely on experts to tell him or her what should go on.

    I really hope this puts some light on where the problems are with the system and where it needs to be fixed. As for "casting doubt" by defense counsel, if two experts disagree, the doubt is there already. For that information to be brought before the jury is what the people who founded our country had in mind. To keep it from them or not tell the defendant, his attorney, or anyone on his side that it existed is just wrong.

    ReplyDelete
  11. I was one of a small handful of police officers attending this seminar. I was also in the latent print breakout group where the ACE-V issue came up. Mr. Strong was clearly caught off guard when the class went from a seminar to a cross examination. It appeared to me that Mr. Strong entered the room intending to simply give a short PowerPoint presentation to a group of novice fingerprint folks. The attendees were preparing their cross examination questions as early as lunch the day before. I am not sure what DPS does or does not do in their lab. I am not even sure that Mr. Strong was in a position to properly respond to the questions. He was clearly caught off guard and not comfortable when he was faced with a large room full of people waiting to hit him with another incriminating statement. There were even a couple of attorneys in the room who were video taping the exchange between Mr. Strong and the audience. I don’t think that DPS properly prepared Mr. Strong for the type of crowd that would be in attendance and I think that he got nervous when the climate started going downhill. The poor guy was on trial from my perspective. I did raise my hand and speak in defense of a couple of simple issues that some in the crowd were trying to distort.

    The latent print and HGN breakout sessions took on a life of their own, in part because many members in the audience are attorneys who are starved for the knowledge as well as any angles to discredit the prosecutions case (what they should do). I saw Lt. Jackson (DPS) before the breakout session started. I gave him a heads up that several members of the audience were working hard to plan a way to trip him up on HGN. To his credit, he obviously came to the seminar expecting such an attack. Lt. Jackson did a great job of delivering his material, maintaining control of the room and properly answering the questions posed to him. Many of the attorneys said that they had been defending DWI cases for 15 years and had never seen what HGN really looked like on video.

    The rest of the seminar was very informative in my opinion. Dr. Siegel did put his opinion on percentages out there. He also said in this seminar that he did not think that individualization was necessary in fingerprint work. He made this clear to the audience.

    I attended the seminar and I did not leave feeling that the host or speakers were on a forensic science bashing campaign. A large percentage of the audience consisted of attorneys. Many of them were very much looking for ways to discredit whatever was being discussed. I am glad that I attended the seminar and would go again if invited. The biggest factor that I saw influencing the attorneys was a lack of understanding in the forensic science studies. They were bringing all sorts of wild accusations up that would have never come up had they been exposed to how forensics works in the real world.

    Like it or not, these same attorneys are going to become the judges of tomorrow. They will be the future gatekeepers of what does and does not get admitted into the courtroom. I watched Lt. Jackson convert many HGN skeptics in the room. He did it by coming to the seminar with materials that allowed the attorneys to see the facts for themselves. The solution to what I saw is a combination of more forensic training available to attorneys and an understanding that they come to the classes to learn, not to trick the speaker into a fumble. Maybe I’m asking for too much. I learned something from the attorneys, judges and the speakers that were at the seminar. All sides made relevant points that need to be considered.

    Dale Bible
    Crime Scene Investigator

    ReplyDelete
  12. "I don't think DPS was intentionally concealing the information"

    I agree with that, 11:38 - Strong didn't even seem to understand what was the concern. I didn't mean to imply there was bad faith and my use of "hide" in response to 11:40 was a poor word choice.

    It's by no means the labworkers' fault and of course DPS has its own counsel before you even get to the AG.

    That said, it would be more probative if they included the information in their report and I think they should in the future. It's highly likely from what we heard there there have been many past cases where examiners disagreed, it wasn't disclosed in the final report, and nobody at trial on either side ever knew about it.

    ReplyDelete
  13. Unfortunately, Mr. Breakfast will not be able to share additional information regarding the supposed 1995 IAI Study, because there never was one. Hopefully he'll do a better job of researching his material next time, but I doubt it.

    ReplyDelete
  14. "Unfortunately, Mr. Breakfast will not be able to share additional information regarding the supposed 1995 IAI Study, because there never was one. Hopefully he'll do a better job of researching his material next time, but I doubt it."

    Maybe you should do a better job of reading the articles first, 7:43. Grits gives a quote from an LA newspaper that cites the 1995 study. Should he rewrite the articles before he quotes them?

    As for the study he is talking about, I've heard Dr. Siegel talk about it on more than one occassion. It's a legitimate study that illustrates problems in fingerprint science. Siegel adds that he does not believe fingerprint science is invalid, just that we have work to do to solve the problems with the field.

    ReplyDelete
  15. Actually somebody forwarded me the study, 7:43, or at least what appears to be the study to which Siegel and the LA Times referred, though I still don't have a link. Shoot me an email if you want a copy.

    It was performed on behalf of the American Society of Crime Lab Directors by an outfit called Collaborative Testing Services Inc. and had 156 responses, out of which 34 had false positive IDs.

    12:52, don't be so hard on 7:43: reading is harder than just making stuff up, just as it's easier being anonymous than accountable. Lower your expectations or you'll find nothing but disappointment in blog comment sections. At least the guy showed enough respect to call me "Mr. Breakfast"!

    ReplyDelete
  16. Here's your link:

    http://www.nasams.org/forensics/for_lib/Index/Fingerprints/Prof%20Tests/index_html

    ReplyDelete
  17. There are several reasons the results of CTS Proficency test should not be considered as a measure of an "error rate" in a forensic testing procedure. If interested please see the following link to a position paper issued by CTS on this issue:

    http://www.ctsforensics.com/assets/news/CTSErrorRateStatement.pdf

    ReplyDelete
  18. The 1995 IAI-CTS test was administered under uncontrolled conditions with no way to validate the qualifications of the participants. If this test had been provided only to certified analyst in accredited labs the results would have been much different.

    ReplyDelete