Counting the Innocent: A discussion of estmates

As the number of exonerated Texans mount, the media and courts have begun to indulge in speculation as to just how many more actually innocent people might be locked up, both in Texas and elsewhere across the country.

The United States has less than 5% of the planet's population but 25% of the earth's prisoners. When you deal in that kind of unprecedented volume, after a while, mistakes inevitably get made.

The debate was set off officially some time ago when SCOTUS Justice Antonin Scalia cited an estimate by Astoria, Oregon District Attorney Josh Marquis, who calculated in an op ed that the system enjoyed a 99.973 percent accuracy rate. In other words, he estimated the wrong person was convicted only .027% of the time.

See this blog post by Marquis explaining his estimate's methodology, in which he took an academic's running tally of exonerations, multiplied by a factor of ten, then compared it to total felony convictions to get this infinitesimal ratio.

If the error rate were that low, said Scalia, "that rate, he said, is acceptable. 'One cannot have a system of criminal punishment without accepting the possibility that someone will be punished mistakenly,' he wrote. 'That is a truism, not a revelation.'”

But what if the error rate were higher - and it might be - when would we reach an unacceptable threshold of wrongful convictions? Scalia to my knowledge hasn't given us that benchmark, but Marquis has written that if the justice system "is hurling innocents into prison at a rate of 2 or 3 percent I would agree that such a system is dangerously flawed."

This spring in the New York Times, Adam Liptak took Marquis' estimate as a starting point for a discussion of how to estimate the number of innocent people behind bars. He focused on the principal statistical conundrum in this debate, that "there is no obvious control group to measure these exonerations against."

In other words, whatever you decide is the right numerator for the number of innocence cases, it's difficult or impossible to know what denominator to compare it to.

That's why yesterday I felt pretty good about the innocence estimate based on Texas death row inmates, whose final convictions were reversed at about a 1.52% rate since the death penalty in Texas was reinstated. Unlike murders or sexual assaults, capital murder cases constitute a finite enough group to supply reliable denominator, which as Liptak says is the hardest part of making a credible estimate.

Since I was reacting in part to his calculations, I sent that piece to Josh Marquis and he responded suggesting this caveat, which I mostly agree with:

Here's the problem with extrapolating numbers of "innocents" based on such relatively small numbers when doing systems analysis. One can't say that Valujet is a deadly airline because they lost 75 passengers out of 1000 flights the month that plane crashed. We can say how dangerous flying is in America if we take ALL flihts in one yeear and divide that by air crash deaths.

That's true, and it's part of the tradeoff for using a smaller dataset to make the estimate: It makes the denominator easier to identify, but the margin of error for any estimate becomes greater.

That's why I especially appreciated Marquis forwarding me a letter he sent to the New York Times in response to Liptak's article, in which he updated his estimate in light of criticisms of his math, essentially using a more narrowly defined denominator:

In a column by Adam Liptak Professor Samuel Gross criticized the method by which Justice Scalia relied upon in determining that wrongful convictions are exceedingly rare. Gross' study published in 2005 listed just under 400 cases of both rape, murder, and a few other felonies which he claimed were exonerations. Since it was my arithmetic that is under challenge I refined my statistics. Using federal statistics for the relevant time period (1989-2003) of total rapes and non-negligent homicides only (not merely felonies) and still allowing for the assumption that Professor Gross under-reported exonerations by a factor of 10, so that there were in fact 4000, not 400 false convictions, the rate of rightful conviction is still 99.25%.

No-one is claiming the justice system is infallible and it can always use tinkering but is it better for 1000 guilty men go free to spare one innocent man? How many innocent victims are acceptable losses for those who criticize my math?

Joshua Marquis
District Attorney
Astoria, OR

The DA's updated estimate that perhaps .75% of people convicted were actually innocent gets us closer to the ballpark of the 1.52% number seen among Texas death row exonerees, especially since, Marquis is correct, both numbers suffer from limits because of their datasets and assumptions.

Applying the .75% figure to Texas' prison population, which is currently around 160,000, give or take, that would mean about 1,200 innocent people are locked up in Texas prisons right now. If my estimate based on death row stats is closer to the truth, double that figure.

Those get to be awfully big numbers pretty quickly, don't they? At what point, I wonder, would Justice Scalia consider the error rate too high?

Thanks, Josh, for generously sharing your views and your updated estimate.