Police AI is bought on minutes saved and audited on who got arrested

Police AI has stopped being a forecast. It is a product list with purchase orders attached, and the useful way to read it is to notice that every one of these tools is bought on one number and audited on a different one. The two are never the same number.

Three deployments and the figures each of them carries:

• Facial recognition: at least 14 people wrongfully arrested in the US, with 13 criminal cases dismissed.
• Predictive policing: Geolitica's forecasts for one New Jersey department were right under 0.5% of the time, 0.6% for robbery and assault and 0.1% for burglary.
• Report writing: Axon's Draft One, an LLM drafting from body-cam audio, is the company's fastest growing product, sold on up to 30 minutes saved per report.

The pattern is in the mismatch. Minutes saved per report, alerts generated, hours of patrol directed: those are production metrics, and they are what appears in a procurement decision. An LLM drafting from body-cam audio produces its minutes-saved figure on day one. Whether the right person was arrested is an accuracy metric, and it shows up later, in a court docket or an audit, paid for by somebody else. Nothing about that is unique to policing, but the cost of the gap here is measured in months of detention rather than in brand sentiment.

Where the numbers are hardest

Facial recognition is where the numbers are hardest. NIST has measured false positive rates up to 100 times higher for Black and Asian faces than for white male faces, and the documented wrongful arrests track that finding. One of this year's cases involved a 50-year-old woman who spent more than five months in jail over crimes committed in a state she says she has never visited.

“I hope you don't think all Black people look alike.”

— Robert Williams, Quoted by NPR and the ACLU

Robert Williams, to a Detroit officer after the first known US wrongful arrest from a face match

Prediction fares no better under audit. When The Markup checked Geolitica's output against what actually happened, the hit rate came in under half of one percent, and the software steered patrols disproportionately toward lower-income Black and Latino neighbourhoods. Chicago decommissioned its gunshot detection system after false positives. Each of these tools kept producing its production metric right up to the day it was switched off.

Fourteen against zero

The clearest comparison available is a natural experiment. More than 20 US jurisdictions, Boston, San Francisco and Pittsburgh among them, have banned police use of facial recognition, and no wrongful arrest from a face match has been reported in a city with an active ban. Fourteen against zero is not a controlled study, but it is the only side-by-side the United States currently has, and there is still no federal law setting accuracy standards or judicial oversight for any of this.

Robots are the visible end of the same list and the smallest part of it. We counted them last week: 1,300 on Chinese streets and one in a New York subway station. The software is where the decisions are. And the shape is the one we measured in marketing this morning, where AI took the tasks with a number attached and stalled at the ones that are judged. In advertising the failure costs a campaign. Here it costs five months.

Informational material, not investment advice. Figures are as published by the ACLU, The Markup, NIST and the vendors themselves; the comparisons between them are ours.