Evaluating criticisms of risk-assessments in bail reform

Risk assessments have always been part of the justice system in some form or fashion. But until recently, there was little if any transparency surrounding them. Instead, they involved some unknown process that goes on in a prosecutor's mind, or a judge's, or a probation officer - even police officers. All of these people assess risk of re-offense and "future dangerousness" every day as part of their jobs. But who can really know on what basis?

That's why outsourcing risk assessments to external, independent, transparent instruments has been so controversial - 1) it's taking a decision making function from humans in the system who may rebel at a perception of decreased power, and 2) the fact of its transparency means a risk-assessment instrument may be criticized in a level of detail that the inner workings of a judge or probation officer's decisions could never be.

Grits has argued that risk assessments play a different role in different parts of the system and that pretrial detention is an area where the benefits likely outweigh the problems. Here's an excerpt (6 min.) from the November episode of the Reasonably Suspicious podcast elaborating on that conversation.


While criticisms of risk assessments have not yet scared me off their use in the pretrial detention context, they have caused me to pay closer attention to arguments from those who think they may be problematic.

Beyond critiques of racial disparities, these academic authors (Upturn's John Logan Koepke and David G. Robinson of Georgetown Law) blame risk assessments for using historical data which may not take into account concurrent reforms enacted alongside risk assessments that can mitigate the risk of pretrial failure (the most effective of which, they say, are repetitive reminder calls and texts). They also criticize agencies for basing risk assessments on data from other jurisdictions,  though they note that those created for a specific jurisdiction often have much smaller datasets. So there's a tradeoff between locality and a robust dataset, the importance of which they undersell just a bit. They recommend the instruments be frequently updated using local data to the greatest extent possible (which is a theme we visited in our podcast segment).

In essence, the concern is that data-based actuarial tools are inherently backward-looking and can't capture the effects of concurrent reforms. For example, "Expanded pretrial services will reduce the risk of failure to appear," but a risk assessment based on pre-expansion data wouldn't capture that. On the other hand, if the instruments are updated and validated periodically, over time that should work itself out.

Finally, in the case of bail reform - at least in the Harris County, Texas context - the maximal harm hypothesized from risk assessments simply doesn't outweigh harms from the status quo of requiring money bail for everyone:

If a risk assessment system is developed on data from before risk mitigating reform, and misjudges a defendant’s true level of risk, such decision-making frameworks might unnecessarily burden defendants who, in reality, do not need to be jailed or to receive as much supervision upon release as the matrix suggests, in order to succeed. This observation is especially relevant given the literature on lower-risk defendants and release conditions. Multiple studies have shown that lower-risk defendants actually succeed on release more often when released without conditions, and that placing conditions of release on lower-risk defendants can actually worsen their odds of success.  

Such a scenario — where defendants are systematically overestimated as riskier than they truly are, leading lower-risk defendants to be subjected to conditions of release that are counterproductive — could perversely sustain an avoidably elevated pre-trial failure rate. In the future, policymakers might look back on the move toward non-financial conditions of release as misguided, and might inaccurately conclude that, despite its ills, a money bail system is the least bad option. As we detailed above, the history of bail reform is one of unintended consequences. The risks we describe in here are not an unimaginable parade of horribles. Instead, they are plausible unintended consequences from today’s bail reforms.

That said, Grits agrees this is the greatest potential problem from implementing risk assessments. In many ways it's a bigger issue than racial disparities, or rather, it's the main reason why racial disparities are important. The impact of assessing low-risk people as medium or high can itself have a criminogenic effect and result almost in a self fulfilling prophecy. IMO that's a great argument for constantly updating and validating risk assessment instruments, but not for avoiding their use - at least in a place like Harris County where nearly all defendants, low-risk or high, would otherwise be held pretrial if they couldn't make bail.

The authors suggest adjusting risk assessment scores when the RA is based on pre-reform data:

jurisdictions that are reforming bail practices should rely on fresh and local data, gathered after their other pretrial reforms have taken root, in order either to build or to calibrate their risk assessment tools. Where prereform data is used, the tools must be numerically adjusted to take account of the helpful impact of reforms. Existing “off the shelf” risk assessment tools, whose predictions assume that defendants still face the same long odds of succeeding outside jail, should not be used without adjustment in jurisdictions where those risks have been mitigated.

In my experience that may be a hard sell. Those adjustments would become highly politicized numbers, and I'm not confident decision makers who spent millions fighting bail reform in the courts would rely on reformers' estimates to make them. Plus, if the reforms work, revisiting and adjusting the RA every couple of years, or even more often, would over time resolve the problem. But it's certainly another argument for continuous evaluation of the risk-assessment instrument.

In the podcast, where we landed was that supplementing RA information with counsel at bail hearings might mitigate disparities because a lawyer (in theory) can communicate a client's particularized circumstance to the court in a way the risk assessment cannot. In the near term, before assessments can be evaluated to take into account the effects of concurrent reforms, that may be the best guard we have against their shortcomings - known and unknown - resulting in negative, unintended consequences.