Saturday, August 06, 2016

Austin PD data SNAFU, shuttered DNA lab signal management failures

Austin cops, like the guy recently caught on video who hurled a local schoolteacher to the ground, are the highest paid in the state. Every budget cycle, including this one, APD shows up to ask the city council for more money for additional patrol slots. But more than additional patrol officers, APD needs to address glaring shortcomings in its ability to analyze evidence, solve cases, and track data about crime.

For example, the Statesman reported (Aug. 5) on why the city's count of DWI statistics keeps changing. The short answer is that none of their data people are trained statisticians but just folks who answered "yes" when asked if they knew Excel and Filemaker Pro. Even so, counting the raw numbers isn't exactly graduate-school statistics. The correct tally of how many DWI accidents APD responded to a given period is basic information they're supposed to report upstream to other institutional actors. There's no reason at all to think such mistakes are limited to DWI numbers.

Some of the problems relate to systems and record keeping, not analysts' knowledge of math: "the statistics unit fails to take basic precautions to ensure the quality of its work, experts told the Statesman. Two crime statistics experts told the paper that police departments should treat their queries like pieces of evidence, logging and tracking them every step of the way." Those are best practices learned from job-related training, not a college statistics course.

Further, "experts say the crime statistics unit appears short of important experience and subject matter expertise. Just one of the four staffers had extensive experience working with databases and advanced statistical models when she was hired to work there."

These are significant management failures. It's a truism that you can't manage what you can't measure. And Austin PD appears incapable of deriving even the most basic data - DWI arrest accident totals - accurately from its system. It's also a management failure from a budgeting perspective. The city has over-invested in patrol and under-invested in non-sworn support.

Similar issues - undertrained staff in technical positions - pervaded the Austin PD DNA lab, which had to close earlier this year to revamp its systems and figure out how many prior cases they screwed up, some of which could result in innocence claims.. The Texas Tribune last week (July 30) provided an update on the clusterf$%#. Here's their assessment of the situation at the lab:
allegations of DNA miscalculation at the crime lab have been a longstanding issue. The June letter from the commission suggested that since 2010, lab analysts have been using expired mixtures and methods that are not “scientifically valid nor supported” by the forensic science community.

Former Austin Police Department crime lab senior forensic scientist Debra Stephens, who was fired in 2012 after filing a complaint with the Travis County District Attorney's Office alleging that some employees weren't qualified and didn't follow federal regulations, said officials have ignored issues in the lab for years.

“It shows that the whole system is flawed. Some of these [analysts] are not qualified, have no science background and are pretending to do forensic science,” she said. “I really think everyone should follow the federally established protocols. But they’re getting away with it.”

Similar concerns were raised in 2010 by Cicely Hamilton, a former DNA lab employee, who later resigned after filing a complaint about some employees ignoring situations involving the contamination of DNA samples.

Robert Paine, director of the Master's of Forensic Science Program at Texas Tech University, said the city and state might ultimately have to kick in funds to help labs hire staff and meet standards.
These aren't the only two areas where important support work gets de-prioritized, as suggested by the 2014 Grits headline, "Failure to hire sufficient, competent staff hindering burglary clearance rates." APD wastes a lot of resources with little practical benefit on burglary cases, while underfunding crime-scene support and detective slots to actually solve the cases.

Bottom line: Pretty much across the board, Austin PD prioritizes its patrol cops and fails to prioritize or sufficiently invest in technical support for those functions. Some of that's because of the city council kowtowing to the union, and some of it's because the chief hasn't prioritized these functions, either in his budgets nor with the attention of top management. This sort of inept bungling is the result: A data system so flawed they can't tally DWI accidents arrests and a DNA lab shut down in significant part because of unqualified analysts and shoddy systems. What a mess!

Make Grits Philosopher King and I'd free up officer time for community policing by taking away other time-consuming duties. For example, disallowing arrests for non-jailable Class C offenses would save time for everybody involved. The Meadows Foundation in Dallas has been pushing to shift first-response duties on mental health calls from police to specialized, medical-led units with a healthcare, not a law-enforcement focus. Finally, Grits has long advocated for shifting to verified response for residential burglar alarms, requiring alarm companies to confirm there's been a burglary before responding. (The burglary clearance rate for residential alarms approaches zero.) In the near term, new spending should focus on shoring up these types of non-sworn staff functions, which have been neglected for many years.

Grits isn't saying "don't invest in public safety." I'm saying the supposed stewards of Austin's public-safety dollars have invested unwisely. This pattern of hiring lots of street cops, overpaying them, failing to invest in non-sworn support staff, then watching those key functions fail, has begun to feel repetitive and sad. Grits used to think there was some hidden agenda behind the pattern. These days I'm more inclined to suspect that the folks running the show at APD simply don't know what the hell they're doing.

CORRECTION: This article originally said APD did not track DWI arrests, but it was DWI-related accidents which weren't correctly documented. Corrected throughout. Grits regrets the error.

24 comments:

Anonymous said...

Grits, best damn description of APD cluster@$% you hit the nail on the head. On a grander scale you described the obama govt.

Anonymous said...

Off the subject, but this MJ article, about lead, is interesting. Lots of academic studies using statistical methods, with links, are referenced. What is the old saying: 'an ounce of prevention is worth a pound of performance'.

Anonymous said...

forgot the link: http://www.motherjones.com/environment/2016/02/lead-exposure-gasoline-crime-increase-children-health

Anonymous said...

Those folks in the "honest man's club" just up the road a piece from Austin have a bit of a CF going on as well.
http://www.wacotrib.com/news/courts_and_trials/federal-judge-suggests-da-reyna-has-conflict-in-twin-peaks/article_27ca5067-14ce-5ddf-a841-372f54c11d83.html

Anonymous said...

Gosh, if only Texas had a "forensic science commission" that, as required by law, investigated allegations of neglect or misconduct inside the crime labs, then we wouldn't have to re-investigate 6 years of incompetence from the APD crime lab management.

If only the lab analysts had come forward with their concerns 6 years ago...

If only there were accountability for APD leaders and Police Chiefs for lying to the public 6 years ago...

Can you smell the sarcasm?

Anonymous said...

What spurred the 2016 investigation by the TFSC?

Anonymous said...

The statewide review of mixture interpretation protocols. The issues identified in that review (including the APD issues) are not unique to Texas. But Texas is still the only state to address them.

Anonymous said...

I think it is safe to say that LE is the most dangerous Terrorist against the public.

Anonymous said...

@6:55-

So, they're performing on-site investigations in ALL Texas labs?
We can, and should, expect MORE of these types of reports?
Why couldn't the TFSC just get the written protocols from the labs, read and interpret them without physically visiting the labs? Seems like this would be a faster means of discerning if a lab's protocols were scientific.

And why was there mention of expired reagents and contamination issues that date back to 2010 (see the most recent TFSC report)? This had to be a greater examination of the lab rather than just a generic "statewide review of mixture interpretation protocols".

Anonymous said...

The June 10 letter from the FSC does not state the reason for the on-site assessment of the APD lab. Seems as if they were sent there for other unreported reasons.

Anonymous said...

There was a review of DNA mixture interpretation protocols from labs in the fall of 2015. The review included a review of cases. The results of that review were presented to the commission earlier this year.

Anonymous said...

Although still outrageous, the APD was unable to consistently provide the number of "drunken driving wrecks" not then number of DWI arrests. The former is slightly more complicated, most likely because they don't code wrecks that way, so they need to make assumptions about what to include, and they did that differently each time they were asked.

Anonymous said...

@1:46

According to Bob Wicoff,...

https://www.blogger.com/comment.g?blogID=8597101&postID=4025689823421545905&bpli=1

...there are 700+ cases statewide being re-investigated, which would suggest that near every DNA lab in Texas is affected, not just APD. And these investigations are ongoing (and will be for awhile). So we should expect to see these same type of letters from the TFSC for every DNA lab they visit - if in fact that was the reason for going to the lab.

Moreover, no case-specific information was mentioned in the June 10 letter. Almost everything mentioned was about Quality Assurance. As a betting woman, I bet there were other reasons for the onsite visit, not just for the DNA mixture protocol clarification.

Somebody aired their dirty laundry.

Anonymous said...

There is a 415 page report (including exhibits) on the TFSC's website regarding the Austin PD DNA section. The history of the statewide mixture review and the specific reasons for the APD audit (which identified other issues unrelated to DNA mixtures) are discussed in the narrative section of the report.

Anonymous said...

7:00 PM -

Having been involved in the mixture review process from its beginning, I'll offer the following comments, fwiw.

What Bob Wicoff's committee is doing is not an "investigation" but a case review. The committee is evaluating requests received from convicted individuals whose convictions involved DNA evidence. The review is to determine if the conviction hinged upon DNA mixtures that were interpreted using the RMP approach, which are candidates for reinterpretation using protocols revised as a result of the statewide review. The requests are generally in response to Brady notices from the various DA's offices. The outcome of the review might be a recommendation to reinterpret in some cases, or a recommendation to not not review in others, for instance if the conviction did not involve mixtures, or did not hinge on DNA, or if the RMP approach was not used. The committee is evaluating the cases from a legal process perspective, not a technical scientific perspective.

So the 700+ case figure can't be interpreted in the way you suggest. All labs have had to update their procedures, and may need to issue corrected reports in some cases. But provided that a lab has whom-heartedly participated in this process, I would not expect that there would be an inspection like the APD inspection.

It should also be remember that APD is one of the very few local PDs in Texas that has its own DNA lab. Most public DNA labs are independent County departments, or operate out of medical examiners offices or universities, which have a higher level of scientific expertise than will be found generally in local PDs.

Again, fwiw.

Anonymous said...

...whole-heartedly...

Anonymous said...

"...In 2010, the ASCLD/LAB assessor either did not review the ST validation study or did not appreciate that the quantity of DNA amplified was an inappropriate way to establish an ST for mixture interpretation. More than one analyst stated the quant-based ST was discussed with an auditor but it is unclear with which auditor this discussion occurred. After the quant-based ST was established in 2010, there does not appear to have been another external review of the ST study until the one conducted in May 2016 by the Commission and ASCLD/LAB..."

Apparently, not negligence

Go figure.

Anonymous said...

415 pages and not one mention of Cecily Hamilton's allegations from 2010. Not one mention of the TFSC's failure to perform an on-site audit of APD in 2010.

Cecily Hamilton gets fired. Everyone else gets a free pass.

Anonymous said...

Cecily Hamilton complained about many things back in 2010.

One of the things she did not complain about in any fashion was the way the lab determined the stochastic threshold.

So, she appears to have been part of the problem.


Anonymous said...

@6:23 dolt-
As an analyst, it is not her responsibility to validate the scientific protocols. She follows them, and reports the problems to the management.
Cecily's employment was terminated in 2010. She would not have been involved with the stochastic threshold interpretation. The Management's problems continued well beyond her termination.
If you had bothered to read the 2016 TFSC Report, "In adopting and continuing the use of a quant-based ST from 2010 to 2016, Technical Leaders (TLs) Jeff Sailus and Cassie Carradine and senior analysts in the APD DNA Lab appear to have [mis]interpreted language from the SWGDAM 2010 Mixture Guidelines..."

"The Lab also defended its SOP despite the fact that the analysts had raised concerns with both TLs that the quant-based ST was ineffective in casework."

If the TFSC had done their duty in 2010, they might have found early on the problems that the analysts were concerned about.

Anonymous said...

So scientists don't need to be knowledgeable about the procedures they use? Wow, I didn't know. That's really helpful. Thanks for clearing that up. I had assumed just the opposite. Because they are, well... scientists. Guess I got mud on my face. I'll let all my scientist friends know. They will be relieved.

Anonymous said...

@8:33-

In forensics, no, you don't have to know at all what you're doing.

Regarding the so-called investigation of APD in 2010, the TFSC report states (even thought the didn't perform an investigation)...

"the investigative team included two subject matter experts: Blake Goertz, the Regional DPS Section Supervisor for DNA (Waco) and Cathy McCord, the Regional DPS Section Supervisor for DNA (Lubbock). The team concluded that: (1) the Lab has extensive procedures in place to minimize contamination; (2) the complainant and investigators were unable to identify a single case of bad science being used in criminal prosecutions; (3) the APD’s DNA training program was sufficient and protocols are in line with national standards; (4) the DNA analyst in question was well-trained and competent; and (5) the Technical Leader’s qualifications were sufficient during the Rangers’ audit and the previous five external audits."

We now know that this 2010 report was BS.

It's sadly just that clear.

Anonymous said...

Hey 9:15

You seem to be an apologist for incompetent analysts.

Are you one?

Anonymous said...

Hey 9:31pm

Germane to the forensic field is the attention to detail.

There is no 9:15 comment.