Tuesday, May 06, 2008

Counting the Innocent: A discussion of estmates

As the number of exonerated Texans mount, the media and courts have begun to indulge in speculation as to just how many more actually innocent people might be locked up, both in Texas and elsewhere across the country.

The United States has less than 5% of the planet's population but 25% of the earth's prisoners. When you deal in that kind of unprecedented volume, after a while, mistakes inevitably get made.

The debate was set off officially some time ago when SCOTUS Justice Antonin Scalia cited an estimate by Astoria, Oregon District Attorney Josh Marquis, who calculated in an op ed that the system enjoyed a 99.973 percent accuracy rate. In other words, he estimated the wrong person was convicted only .027% of the time.

See this blog post by Marquis explaining his estimate's methodology, in which he took an academic's running tally of exonerations, multiplied by a factor of ten, then compared it to total felony convictions to get this infinitesimal ratio.

If the error rate were that low, said Scalia, "that rate, he said, is acceptable. 'One cannot have a system of criminal punishment without accepting the possibility that someone will be punished mistakenly,' he wrote. 'That is a truism, not a revelation.'”

But what if the error rate were higher - and it might be - when would we reach an unacceptable threshold of wrongful convictions? Scalia to my knowledge hasn't given us that benchmark, but Marquis has written that if the justice system "is hurling innocents into prison at a rate of 2 or 3 percent I would agree that such a system is dangerously flawed."

This spring in the New York Times, Adam Liptak took Marquis' estimate as a starting point for a discussion of how to estimate the number of innocent people behind bars. He focused on the principal statistical conundrum in this debate, that "there is no obvious control group to measure these exonerations against."

In other words, whatever you decide is the right numerator for the number of innocence cases, it's difficult or impossible to know what denominator to compare it to.

That's why yesterday I felt pretty good about the innocence estimate based on Texas death row inmates, whose final convictions were reversed at about a 1.52% rate since the death penalty in Texas was reinstated. Unlike murders or sexual assaults, capital murder cases constitute a finite enough group to supply reliable denominator, which as Liptak says is the hardest part of making a credible estimate.

Since I was reacting in part to his calculations, I sent that piece to Josh Marquis and he responded suggesting this caveat, which I mostly agree with:
Here's the problem with extrapolating numbers of "innocents" based on such relatively small numbers when doing systems analysis. One can't say that Valujet is a deadly airline because they lost 75 passengers out of 1000 flights the month that plane crashed. We can say how dangerous flying is in America if we take ALL flihts in one yeear and divide that by air crash deaths.
That's true, and it's part of the tradeoff for using a smaller dataset to make the estimate: It makes the denominator easier to identify, but the margin of error for any estimate becomes greater.

That's why I especially appreciated Marquis forwarding me a letter he sent to the New York Times in response to Liptak's article, in which he updated his estimate in light of criticisms of his math, essentially using a more narrowly defined denominator:
In a column by Adam Liptak Professor Samuel Gross criticized the method by which Justice Scalia relied upon in determining that wrongful convictions are exceedingly rare. Gross' study published in 2005 listed just under 400 cases of both rape, murder, and a few other felonies which he claimed were exonerations. Since it was my arithmetic that is under challenge I refined my statistics. Using federal statistics for the relevant time period (1989-2003) of total rapes and non-negligent homicides only (not merely felonies) and still allowing for the assumption that Professor Gross under-reported exonerations by a factor of 10, so that there were in fact 4000, not 400 false convictions, the rate of rightful conviction is still 99.25%.

No-one is claiming the justice system is infallible and it can always use tinkering but is it better for 1000 guilty men go free to spare one innocent man? How many innocent victims are acceptable losses for those who criticize my math?

Joshua Marquis
District Attorney
Astoria, OR
The DA's updated estimate that perhaps .75% of people convicted were actually innocent gets us closer to the ballpark of the 1.52% number seen among Texas death row exonerees, especially since, Marquis is correct, both numbers suffer from limits because of their datasets and assumptions.

Applying the .75% figure to Texas' prison population, which is currently around 160,000, give or take, that would mean about 1,200 innocent people are locked up in Texas prisons right now. If my estimate based on death row stats is closer to the truth, double that figure.

Those get to be awfully big numbers pretty quickly, don't they? At what point, I wonder, would Justice Scalia consider the error rate too high?

Thanks, Josh, for generously sharing your views and your updated estimate.


Anonymous said...

Your math is good. Let's go forth. What about all the Texas inmates that agreed to a plea? The plea bargain eliminates most appeals and consequently most findings of acctual innocents.

With a few more educated guesses, the numbers can easily reach 10% or more. Now we're talking 16,000 and the cost to the individuals and the State are enormous.

Plea bargains may save on Court costs but they are far more costly to individuals and societies on the back end!

Anonymous said...

If you use the true result with arbitrarily raising it by an order of magnitude the exoneration rate is .00075

Anonymous said...

Hey 12:20pm, if you really think 10% of the people that plea guilty are actually innocent then I have a bridge you might be interested in buying...

Gritsforbreakfast said...

Marquis is not estimating an "exoneratinon rate," 12:25, he's estimating an "error rate," or a he would put it, an "accuracy rate," and quite reasonably assuming not every error is captured. The 10x factor, though, is certainly the weakest part of his calculation, which is partially why I looked at the Texas DP stats.

Gritsforbreakfast said...

FWIW 12:27, here's a past discussion by and about attorney bloggers on the rate of their clients who plead guilty who they think are innocent - WIDE range in guesstimates.

Anonymous said...

Thost false guilty pleas where when the state was offering immediate jail release and "time served" on a low level offense.

That is quite different than a false guilty plea that sends you to prison.

Anonymous said...

I think that 12:59 has a valid point. My recollection of the earlier discussion was that people were willing to plea in order to get out of jail not to go to prison. Although I have heard defense attorneys say their client pleaded guilty because they knew they would be promptly paroled from prison and that was the low cost option.

A 1% error rate for the CJ system is not bad for a non DP case. I would like to see a lower rate for DP cases.

kbp said...

Factor in few (if any) care about the finding out who are wrongly convicted serving <10-20 years and how much more stringent on the rules are the courts in cases that are >10-20 years.

I doubt we'll ever know of an sound method to determine how many were innocent.

Anonymous said...

The thing about plea bargains is that they go on your record and eventually you end up in prison because of your record, not because of a crime you actually committed. Actual innocense should be a much greater emphasis of our justice system. Instead, conviction rates rule.

My point - plea bargains are costly on the back end for both individuals and societies stands!

Anonymous said...

My point - plea bargains are costly on the back end for both individuals and societies stands!

You are mistaken because 95% of the cases plea out. Most of the remainders are dismissed, and only a tiny few proceed to trial.

If there were no plea bargins then what to do with the 19 out of every 20 cases currently disposed of with a guilty plea? Not only would it would require building lots more courthouses and jails, it also means hirinbg tons more judges and prosecutores. Also since policemen would be testifying against everyone they arrested, you're going to massively increase the size of the police force.

The more I think about the idea of getting rid of plea bargins, the worse it seems.

Anonymous said...

Those DP cases ought to have the lowest numbers, but instead have the highest numbers. In fact it seems to be justice of less than three sigmas (~2.37 sigma). Industry currently tries for 6 sigma in their efforts but the courts can't do even 3 when a life is on the line. What does that say about the American justice system? Three sigma would yield less than 27 reversals per 10K sentences, four less than 64 per 100K, five less than 5 per 10M, and six less than 2 per billion. While the rest of us have been producing higher quality goods our court system has been producing inferior justice. It time we tried for six sigma in our justice system. Judges can't judge well once they become stuck in a particular spot. They begin to know and believe the prosecutors even when they are bringing in lying witnesses or withholding evidence. And lawyers get off with the minimum if anything - see how often one goes to jail if he's caught on a DWI/DUI. The familiarity with the prosecution and the absence of real penalties on lawyers/police/CPS for violating the law is killing the justice system.

I suggest rotating judges between courts and making prosecutors, lawyers, cops, and CPS workers accountable by creating a special court of justice that prosecutes them and allows them to be sued for their acts.

For instance I know of a case where the cop testified that he was waiting to turn left into his parking lot on a dark moonless night for two vehicles to go by. after they went by, he pulled into the lot and heard a vehicle gun his engine and saw the second vehicle pass the first in a no passing zone. Now despite the fact there was a fence, low lying vegetation, signs, a rise in the road, more than a couple of football fields and the fact his vision was extremely poor from having just watched those same vehicles come toward him with their headlights on he claimed he could see the violation. Despite the defendant claiming he had just turned on to the road at a spot well after this point and bringing sworn dispositions testafying that he wasn't on that road at that spot, the judge convicted the defendant. I think that if this case had gone before a jury instead they would never have believed that tale.

SB said...

Of course innocent people take a plea. As 1st times in the system they are lost and very much afraid. The promise that an example will be made out of those who take their cases to trial has impact. Being told your Judge goes off the deep end adds to the fear. Everyone from the arresting officer forward has determined guilt. A jury of peers is a group of people from a society that is known for locking folks up. A plea can start looking attractive to an innocent person. When the panic wears off it becomes evident it isn't a bargain.If there is a next time it's a whole different story.

Gritsforbreakfast said...

I am TOTALLY down with a "six sigma" standard as an error rate. I think that analogy is dead on.

I also like the idea of rotating judges among courts, which would address the very real problem mentioned earlier by sunray's wench.

FWIW, I don't think I've ever heard anybody propose "getting rid of plea bargains," and don't think that result's implied by these data or anything I've written.

I do think that plea bargain's are inappropriately controlled by contract law, which does not account for their coercive nature, but just because I think the process should be reformed doesn't mean I think pleas should be eliminated.

JSN, a 1% error rate is not bad unless you're the lucky one out of 100! The number Scalia thought was okay was 27 times lower than Marquis' updated estimate. I'm not sure even Scalia would think a 1% error rate is acceptable.

I've always thought, FWIW, that many innocents convicted likely wind up on probation, since that's often what happens with weak cases. I agree with the first commenter the rate may be even higher for probation and time-served plea bargains. But frankly, if it were 10% or more, it'd be time to stop blogging and take up arms! That'd be a Kafka-esque, totalitarian nightmare that honestly I don't believe we're even close to right now.

That said, while I don't think the number of wrongful convictions is remotely as high as 10%, neither do I find a 1% rate acceptable.

Thanks, folks, for a good discussion. (REALLY groovin' on the six sigma idea!) best,

Anonymous said...

"that rate, he said, is acceptable. '

That statement sickens me. What are people to him, cattle? I deplore utilitarianism in the application of law. I don't find the conviction of 1 innocent person to be acceptable. How can Scalia, purportedly a practicing Catholic, say such a thing?

"If there were no plea bargins then what to do with the 19 out of every 20 cases currently disposed of with a guilty plea? Not only would it would require building lots more courthouses and jails, it also means hirinbg tons more judges and prosecutores. Also since policemen would be testifying against everyone they arrested, you're going to massively increase the size of the police force."

The fact that more judicial resources might be needed to give real justice is no reason, to me, not to do so. Plea bargains are, in my opinion, a perversion of the system and an encouragement for bearing false witness.

The thing is, I think that the criminal justice system should be reserved for those crimes which actually involve victims. I believe many cases such as criminally negligent homicide or white collar fraud should emphasize restitution for victims (other than through insurance) and not incarceration.

I guess that I am more of a Blackstone/ Old Testament kind of gal than a Benthamite.

Anonymous said...

I dunno. I think an estimate based on known wrongful convictions in capital cases would be on the low side.

Years ago I researched the California Penal Code on various kinds of theft for a paper I was writing. The thickest section of the Calif Penal Code was capital punishment. And I mean by far.

On the whole it attempted to define the correct application of the death sentence down to the jot and tittle, especially procedures. The feeling I got as a naif reading law was that the powers that be in the state had an awful conscience about it, so got bogged down in details to ensure it was done right if done at all.

I've never checked the Texas Penal Code, but I'd bet without looking at my hole cards that its section on capital punishment is also the thickest part of the code. Maybe not.

Anyhow, of all sentences for all crimes, it is the most carefully scrutinized. So my guess is that the error rate for capital punishment is much lower than other crimes. If you get 1.5 percent, 10 percent generally isn't a stretch.

Anonymous said...

Grits - can you identify anyone who has been executed who was subsequently found to be innocent? List him/her or them. Thanks,


kbp said...

...subsequently found to be innocent?

They have investigators and courts set up to find them "innocent"?

Gritsforbreakfast said...

Plato, there has never been a definitive finding of innocence (e.g., through DNA) AFTER an execution. If DNA evidence were routinely saved historically, though, IMO we'd already have seen a definitive case.

However, there are several cases where most people besides the prosecutor think they were innocent of the crime they were executed for.

The one I find personally most compelling is the story of Cameron Todd Willingham, see this Chicago Tribune story.

There's another recent case in San Antonio, Ruben Cantu, where the sole witness recanted and claimed to have been coerced, and the evidence looks pretty strong he didn't do it.

There are several others over the years where strong actual innocence claims can be made and the only honest conclusion one could reach, at the end of the day, IMO, is "I don't know." Somebody who focuses more on the death penalty like Steve Hall or Scott Cobb might be able to provide more examples.

Anonymous said...

Let us say there are three type of CJ system errors.
1. A guilty person is freed..
2. An innocent person is convicted.
3. An innocent person pleads guilty.

I doubt the error rate for type 1 is smaller than 10% and I doubt the type 2 error rate is smaller than 1% and I have no idea what the type 3 error rate is (no doubt there are some persons don't think they occur).

What I should have said was if the error rate were as small as 1% it would be a good thing.

TCADP said...

Anytime we execute someone, an error rate of any % is unacceptable.

See one of the many showings of At the Death House Door hosted this month by TCADP, www.tcadp.org to see why. (Imagine telling the family, oops we executed your son on accident.)

There will also be a hearing on wrongful conviction hosted by Sen Ellis in the Senate Chamber on Thursday from 1:00 to 5:00 for any who are interested.

Toxic Reverend said...

Advocates had been using web pages such as

"FBI Witch Hunt"
(Censored version at the Archives Wayback Machine),
to document the abuses of the judicial system against advocates
with grand juries and other government harassment.

The original web page has been "compromised", but the
Archives still has copies of what had been posted there:


FBIwitchHunt.com was created in response to the US
government's increasing use of Grand Juries, and other tools
of repression, against activists. The government relies
heavily on misinformation and secrecy in its efforts to crush
various social and political movements. It is our goal to
provide a forum for activists across the country, from all
movements, to relay information about their ongoing
struggles against government repression, and to provide
some insight into how to combat this repression.

End of excerpt.




Green Scare
"Operation Backfire" is the name given by the Federal Bureau of Investigation (FBI) to the ten year old investigation and indictment of fifteen individuals accused of several arsons in the Pacific Northwest between 1996 and 2001 claimed by the Earth Liberation Front (ELF) or the Animal Liberation Front (ALF). It is because of these arrests and the fervor that federal agencies are seeking to impose severe sentences under the statutes of "anti-terrorism" that the term, "Green Scare" has resurfaced and is being used to describe the draconian measures taken against environmental activists in the same way that alleged communists were targeted in the McCarthy-era Red Scare.

These indictments have become the subject of much debate since they were based primarily on the testimony of a heroin addicted self-professed serial arsonist, whose name apparently came up in a grand jury inquisition of a bitter ex-girlfriend.

End of excerpt from;

"Operation Backfire"
Green Scare

From another perspective, there is an absence of accountability with corporate homicides, such as Firestone tires and the like.

In reference;


The Next Wave – Call It Red Collar Crime
20 Corporate Crime Reporter 32(1), August 3, 2006


Twenty Things You Should Know About Corporate Crime
21 Corporate Crime Reporter 25, June 12, 2007


Number 20

Corporate crime inflicts far more damage on society than all street crime combined.

Whether in bodies or injuries or dollars lost, corporate crime and violence wins by a landslide.

The FBI estimates, for example, that burglary and robbery – street crimes – costs the nation $3.8 billion a year.

The losses from a handful of major corporate frauds – Tyco, Adelphia, Worldcom, Enron – swamp the losses from all street robberies and burglaries combined.

Health care fraud alone costs Americans $100 billion to $400 billion a year.

The savings and loan fraud – which former Attorney General Dick Thornburgh called "the biggest white collar swindle in history" – cost us anywhere from $300 billion to $500 billion.

And then you have your lesser frauds: auto repair fraud, $40 billion a year, securities fraud, $15 billion a year – and on down the list.

Number 19

Corporate crime is often violent crime.

Recite this list of corporate frauds and people will immediately say to you: but you can't compare street crime and corporate crime – corporate crime is not violent crime.

Not true.

Corporate crime is often violent crime.

The FBI estimates that, 16,000 Americans are murdered every year.

Compare this to the 56,000 Americans who die every year on the job or from occupational diseases such as black lung and asbestosis and the tens of thousands of other Americans who fall victim to the silent violence of pollution, contaminated foods, hazardous consumer products, and hospital malpractice.

These deaths are often the result of criminal recklessness. Yet, they are rarely prosecuted as homicides or as criminal violations of federal laws.

End of excerpts from:
Twenty Things You Should Know About Corporate Crime
21 Corporate Crime Reporter 25, June 12, 2007


Easter Sermon 08:
Red Collar Criminals Playing American Roulette With Our Lives


1: Trade Secrets,
A web page from Bill Moyers Public Broadcasting show, national television.
Documentation of over 50 years of corporate cover up, costing an unknown
number of American lives, with out criminal presecution.

2: PDF Skeletons in the Courthouse:
Hazards to the Public Remain Secret.
April 2002 By Coalition for Consumer Rights,
A copy of the "censored" PDF report is re-posted at
I had download the report and made back ups . So I and re-posted it.
Their web page is gone from the net and the times that I have
tried to call the authors (lately) their phone is "being checked for trouble".

W.R. Grace to Pay $250 Million for Libby Cleanup,
by Liz Borkowski
A 10-count criminal indictment against seven
senior current and former Grace officials alleging
conspiracy. For knowing endangerment, obstruction
of justice and wire fraud for endangering the
people of Libby by concealing the well-documented
hazards of the tremolite asbestos.

In reference to the above story, the criminal
charges are "pending". National support is needed
to see that these criminal charges against the
individuals responsible are in fact heard in a court
of criminal (not civil) court.

The charges have been brought by
William W. Mercer
United States Attorney
for the District of Montana

Tom Krohmer
Environmental Technologist
The Toxic Reverend

Anonymous said...

With some of the Judges who occupy seats in District Courts now, it is no wonder people accept a plea deal. Some are so mean and inconsiderate of the lives they are ruinging no wonder there are so many plea deals. The could be the lesser of the two evils, face a crooked DA and Judge and you will lose and if you elect to take a plea deal, you lose the right to appeal.

The CCA will not do the right thing and read both Writs submitted by the Defense attorney and the State, if the Judge signs with the State, they automatically rule against an appeal.

Now is the time to start cleaning out the court system and get rid of the Judges in the District Courts and the Judges who sit on the CCAs. Vote against all incumbents and clean out part of the mess we have in our court system.

Toxic Reverend said...

A classic "But First Moment";
Before the judges can be voted out, campaign finance laws will most probably need to change.

Example in this news article;

Justice for Sale - JS Online:
Poll says money, courts atroubling mix
... court, and 26% of judges agree, according to the survey, conducted for Justice at Stake, a national non-partisan organization concerned with judicial issues. ... By DAVID DOEGE, of the Journal Sentinel staff, Last Updated: Feb. 17, 2002

Anonymous said...

Let Scalia or zealous prosecutors sit in prison and then we can really get down to talking about an "acceptable rate".