Monday, November 20, 2017

Evaluating criticisms of risk-assessments in bail reform

Risk assessments have always been part of the justice system in some form or fashion. But until recently, there was little if any transparency surrounding them. Instead, they involved some unknown process that goes on in a prosecutor's mind, or a judge's, or a probation officer - even police officers. All of these people assess risk of re-offense and "future dangerousness" every day as part of their jobs. But who can really know on what basis?

That's why outsourcing risk assessments to external, independent, transparent instruments has been so controversial - 1) it's taking a decision making function from humans in the system who may rebel at a perception of decreased power, and 2) the fact of its transparency means a risk-assessment instrument may be criticized in a level of detail that the inner workings of a judge or probation officer's decisions could never be.

Grits has argued that risk assessments play a different role in different parts of the system and that pretrial detention is an area where the benefits likely outweigh the problems. Here's an excerpt (6 min.) from the November episode of the Reasonably Suspicious podcast elaborating on that conversation.

While criticisms of risk assessments have not yet scared me off their use in the pretrial detention context, they have caused me to pay closer attention to arguments from those who think they may be problematic.

Beyond critiques of racial disparities, these academic authors (Upturn's John Logan Koepke and David G. Robinson of Georgetown Law) blame risk assessments for using historical data which may not take into account concurrent reforms enacted alongside risk assessments that can mitigate the risk of pretrial failure (the most effective of which, they say, are repetitive reminder calls and texts). They also criticize agencies for basing risk assessments on data from other jurisdictions,  though they note that those created for a specific jurisdiction often have much smaller datasets. So there's a tradeoff between locality and a robust dataset, the importance of which they undersell just a bit. They recommend the instruments be frequently updated using local data to the greatest extent possible (which is a theme we visited in our podcast segment).

In essence, the concern is that data-based actuarial tools are inherently backward-looking and can't capture the effects of concurrent reforms. For example, "Expanded pretrial services will reduce the risk of failure to appear," but a risk assessment based on pre-expansion data wouldn't capture that. On the other hand, if the instruments are updated and validated periodically, over time that should work itself out.

Finally, in the case of bail reform - at least in the Harris County, Texas context - the maximal harm hypothesized from risk assessments simply doesn't outweigh harms from the status quo of requiring money bail for everyone:
If a risk assessment system is developed on data from before risk mitigating reform, and misjudges a defendant’s true level of risk, such decision-making frameworks might unnecessarily burden defendants who, in reality, do not need to be jailed or to receive as much supervision upon release as the matrix suggests, in order to succeed. This observation is especially relevant given the literature on lower-risk defendants and release conditions. Multiple studies have shown that lower-risk defendants actually succeed on release more often when released without conditions, and that placing conditions of release on lower-risk defendants can actually worsen their odds of success.  
Such a scenario — where defendants are systematically overestimated as riskier than they truly are, leading lower-risk defendants to be subjected to conditions of release that are counterproductive — could perversely sustain an avoidably elevated pre-trial failure rate. In the future, policymakers might look back on the move toward non-financial conditions of release as misguided, and might inaccurately conclude that, despite its ills, a money bail system is the least bad option. As we detailed above, the history of bail reform is one of unintended consequences. The risks we describe in here are not an unimaginable parade of horribles. Instead, they are plausible unintended consequences from today’s bail reforms.
That said, Grits agrees this is the greatest potential problem from implementing risk assessments. In many ways it's a bigger issue than racial disparities, or rather, it's the main reason why racial disparities are important. The impact of assessing low-risk people as medium or high can itself have a criminogenic effect and result almost in a self fulfilling prophecy. IMO that's a great argument for constantly updating and validating risk assessment instruments, but not for avoiding their use - at least in a place like Harris County where nearly all defendants, low-risk or high, would otherwise be held pretrial if they couldn't make bail.

The authors suggest adjusting risk assessment scores when the RA is based on pre-reform data:
jurisdictions that are reforming bail practices should rely on fresh and local data, gathered after their other pretrial reforms have taken root, in order either to build or to calibrate their risk assessment tools. Where prereform data is used, the tools must be numerically adjusted to take account of the helpful impact of reforms. Existing “off the shelf” risk assessment tools, whose predictions assume that defendants still face the same long odds of succeeding outside jail, should not be used without adjustment in jurisdictions where those risks have been mitigated.
In my experience that may be a hard sell. Those adjustments would become highly politicized numbers, and I'm not confident decision makers who spent millions fighting bail reform in the courts would rely on reformers' estimates to make them. Plus, if the reforms work, revisiting and adjusting the RA every couple of years, or even more often, would over time resolve the problem. But it's certainly another argument for continuous evaluation of the risk-assessment instrument.

In the podcast, where we landed was that supplementing RA information with counsel at bail hearings might mitigate disparities because a lawyer (in theory) can communicate a client's particularized circumstance to the court in a way the risk assessment cannot. In the near term, before assessments can be evaluated to take into account the effects of concurrent reforms, that may be the best guard we have against their shortcomings - known and unknown - resulting in negative, unintended consequences.


Anonymous said...

Consider how the state uses the Static 99... The test guidelines are ignored because it generates a "useful" result as opposed to an "accurate" one.

Risk assessments can probably be very useful, but the precedent so far is that even when they are mandated there is no accountability on how they are employed.

Gritsforbreakfast said...

The use case for bail RAs is a lot different from using the Static 99 for sex offenders on the back end. Completely different set of issues in play.

Anonymous said...

I think the issue is that you can't just rely on local "due process" functionaries to properly apply tools without oversight.

The Static 99 is used on all female sex offenders in the state in spite of the text instructions being explicit that it cannot evaluate females.

In a bond risk assessment scenario, you're either relying on a vague set of judgment guidelines which could be pushed to the lowest eligible decision maker (read: hired not elected) who can be easily steered to bias their decisions to reflect Management's position on who should or should not be incarcerated OR you have an explicit set of easy, judgement agnostic guidelines (like the Static 99, 10 simple yes or no questions) that gets applied selectively because no one is able to hold the evaluation administrator accountable.

For bond the defendant is already in jail, so if a county official rubber stamps the evaluation to keep you in jail you're still in the same position as you are today... Waiting for a hearing, paying an attorney. Judges may chastise the test proctor for simply not completing enough "get out of jail" worksheets, but then it's just an HR dilemma, did you follow procedure as determined by... Your supervisor?

What I'm saying is the state is already rubber stamping "dangerousness" assessments, so you'd do well to preempt it in the future.

What's stopping the bond industry from promoting "stay in jail with bond" assessments in the magistrate courts?

If risk assessments are merely "due process" and not a "fair hearing," then we're still losing.

Anonymous said...

The best predictor of future risk? Past behavior.

mike connelly said...

Thank you for this. One of the most frustrating things about the assessment critiques is that critics have been comparing to a world where the offender is judged completely neutrally and "correctly" which the assessments will now screw up. Your take on the assessments that have been done for decades in the idiosyncratic heads of the judges, DAs, defense, P&P folks, etc. without transparency or accountability is exactly right. That's the world that has gotten us to the unfortunate state we're in, not risk assessment tools. There are dangers to the use of those tools, but, since we know them and can in theory require the changes and updatings that you recommend, we can harness their benefits and avoid their costs far better than we could with traditional systems.

You are also right about the potential problems of assessment politicization. I heard a victims' advocate in SC talk about how she had studied these assessments people were talking about for pre-sentence reforms and how she disagreed with them and how she and her colleagues might just have to come up with their own. We seriously think that the DAs and defense attorneys who have questioned the use of these tools for sentencing purposes wouldn't propose their own or make political hay out of what they see as problems with variables used or factors they see as missing? The reform groups that advocate use of assessments based on aggregates in getting individual sentences "right" need to speed dial you for some input. And then listen to it.

Anonymous said...

There is a fundamental lack of understanding that "public safety" is defined at the local level. Consequently, a centralized push out of any risk assessment tool for bond, probation, parole etc. will be subject to the manipulation of that locale. This is most evident in jurisdictions where criminal justice officials serve a populace that votes their fear of being a victim of crime. Data Driven Evidence Based Practice is no consolation to voters whose fears are legitimized every time their lock'em-all-up-at-all-cost candidate broadcasts their "fear crime campaign" and gets re-elected. Some local systems embrace change and look forward to better outcomes and some will oppose change because they are rewarded by the electoral process for not changing.

Anonymous said...

Whatever risk assessment they were using in Smith County probably needs to be reevaluated after the events of Thanksgiving afternoon.