Don't believe the headlines on the Harvard study purporting to show racial bias does not exist in police shootings

Racial bias in policing has emerged as one of the most contentious issues of our time. The issue has been difficult to study due to gaps in data, but this week a Harvard economist released a report that used statistical analyses to determine whether racial bias plays a role in police decisions on use of force. While news headlines covering the study declared no racial bias exists in officer-involved shootings, what the study really shows is the limitations of data and statistical analyses in getting to the bottom of the issue.

On Monday the New York Times covered the report in its Upshot column with an article, “Bias is Found in Police Use of Force but Not in Shootings.” The New York Times headline overstates the study’s findings, and Matt Drudge, Rush Limbaugh, and Rudy Giuliani, among others, picked up the story to conclude that the Black Lives Matter movement is premised on a bunch of hogwash. But the study is actually much more limited than the Times’s misleading headline, and it’s wrong for the author or media to conclude from its findings that bias does not exist in police shootings in the United States. Upshot yesterday published a Q&A with Dr. Fryer addressing the limitations of his report, and this follow-up included a much more measured description of the study’s conclusions. And today the New York Times Public Editor addressed reader complaints with the original Upshot article.

The main weaknesses in the study revolve around its limited data collection and overly broad interpretation, and its failure to take into account how racial biases impact policing and police reporting. While the author offers a series of disclaimers throughout the report recognizing these limitations, the sweeping conclusions he still draws – and certainly the conclusions media have drawn – seem to ignore those important limitations.

The piece of the study dealing with officer-involved shootings contains two parts. In the first part, the author used officer-involved shooting reports from ten different police departments to ask one question: whether race was a significant factor in “whether or not the officer shoots the suspect before being attacked.” To answer this question, the author studied incidents where an officer shot a suspect after being attacked or having a weapon pulled or attempted to be pulled on the officer, and compared those shootings to shootings where the officer was not attacked or threatened in these ways.

But the author’s data is flawed because it relied entirely on the officer’s own account of whether or not the officer was attacked or threatened, and no attempt was made to independently verify the level of threat as described by the officer. The author admits this weakness: “police departments … may contain police officers who present contextual factors at that time of an incident in a biased manner.” In other words, reports submitted by law enforcement may not be accurate reflections of the actual level of threat. This could occur through deliberate misreporting. It could be poor memory.  But it could also be the result of implicit racial bias, which (as other research has shown) causes individuals to perceive people of color as more threatening than identically positioned white people, which in turn may lead the officer to inaccurately report force-justifying behavior by black individuals. Since I began assembling data on all officer-involved shootings in Texas since 2015, I have seen numerous police accounts that describe the presence of a weapon or other threat, which is later disputed by witness or media accounts. Any quantitative study regarding police shootings needs to start with independently verified data, such as body camera footage.

In the second part of the study, the author used data provided by the Houston Police Department to come up with a dataset that purports to be comprised of incidents where an officer may have used lethal force, but did not. The author then compared that dataset, in which such force was not used, to the Houston dataset where the officer did shoot, to ask whether race was a significant factor in the decision to use potentially lethal force.

But the study does not acknowledge the barriers to extrapolating from this geographically limited data. Although headlines such as “Bias is Found in Police Use of Force but Not in Shootings” (New York Times) or “No Racial Bias in Police Shootings, Study by Harvard Professor shows” (Washington Times) would lead you to believe that its findings apply nationwide, this dataset dealt only with Houston, and indeed, only with the Houston Police Department.

And even just limiting the conclusions to Houston, I'm skeptical of isolating a dataset of incidents in which a shooting could have occurred but did not.  An economist at Columbia University was also skeptical and particularly troubled by the fact that the arrestee pool looked pretty different from the shooting victim pool.  For example, 18% of arrestees were female, but only 4% of shooting victims were female.

Neither part of the officer-involved shooting study takes into account a key stage where racial bias has been found pervasive: in decisions on whether to conduct traffic stops, searches or other discretionary policing. The author recognizes the role racial bias may play in the initial interaction, but this limitation in the study deserves more than the one-sentence disclaimer that appears. Police shootings do not occur in a vacuum, and some of the most shocking killings by police have been the result of this sort of discretionary policing that disparately impacts communities of color. Recall that Eric Garner was being arrested for selling cigarettes, Michael Brown was being ordered to get out of the road, and Philando Castile was pulled over because the officer thought he resembled a robbery suspect with a “wide-set nose.”

An underlying problem I see with the study is the basic assumption in economics that people take into account all available information, probabilities of events, and potential costs and benefits to act consistently in choosing the self-determined best course of action. This “rational actor” assumption does not leave room for implicit biases or other explanations for behavior that are not determined by conscious, rational decision-making. So, the author admits that “the penalties for wrongfully discharging a lethal weapon in any given situation can be life altering, thus, the incentive to misrepresent contextual factors on police reports may be large.” But yet the author attributes his finding that racial differences do not impact the decision to use lethal force to the fact that “officers face discretely higher costs for officer-involved shootings relative to non-lethal uses of force.” By assuming police behavior is the result of known incentives, the author fails to acknowledge how implicit biases affect perceptions in police/civilian interactions. And when an officer is considering whether a person is a threat, the way the officer perceives another person can mean everything.