The Flawed Promise of Crime-Prediction AI: Lessons from the UK's Police Experiment
In recent years, artificial intelligence (AI) has become an increasingly integral part of law enforcement, with police forces around the world experimenting with predictive analytics to identify potential crimes before they occur. One of the most ambitious and controversial examples of this trend is the UK's Avon and Somerset Police, which developed a sprawling predictive analytics program. However, an investigation by WIRED and partner organizations has revealed significant flaws in the system, raising questions about the reliability and ethical implications of AI in policing.
The Think Family Database: A Controversial Experiment
At the heart of Avon and Somerset Police's predictive analytics program is the Think Family Database, a massive repository of sensitive information about nearly half a million people in the city of Bristol. Launched in 2016, the database was a collaboration between the police and Bristol City Council, aimed at creating a 'picture of threat, harm, and risk' in the region. It included data such as police intelligence reports, housing status, mental health records, teenage pregnancies, and enrollment in parenting courses.
Using this data, officials developed machine-learning models to assign risk scores to individuals, with the goal of predicting who was most likely to commit crimes or become victims. For example, the Child Sexual Exploitation (CSE) model analyzed factors like school absences, mental health concerns, and social connections to identify children at risk. However, the system faced criticism almost immediately, with researchers pointing out that the variables used could act as proxies for poverty, potentially leading to bias.
Ethical and Practical Challenges
Despite the initial enthusiasm for predictive policing, the Avon and Somerset program faced numerous challenges:
-
Lack of Transparency: The public was largely unaware of the program's existence, and even those who were informed had little understanding of how their data was being used. This lack of transparency undermined public trust.
-
Bias and Inaccuracy: Several models, including those for predicting child sexual exploitation and criminal exploitation, were quietly abandoned after officials deemed them unreliable. For instance, one model was found to produce 'genuinely poor predictive performance,' with fewer than one in 10 flagged individuals actually committing crimes.
-
Function Creep: Over time, the systems became more expansive, combining more data and spreading beyond their original purposes. This raised concerns about overreach and the potential for misuse.
-
Legal vs. Legitimate Data Use: While the program relied on 'legal gateways' to collect data without explicit consent, critics argued that legality does not equate to legitimacy. The lack of public engagement and trust further compounded this issue.
The Human Impact
The investigation also highlighted the real-world consequences of flawed predictive systems. John Pegram, a Bristol resident, discovered he was included in the Offender Management App, a tool used to profile 'dangerous criminals.' Despite having no recent offenses, Pegram was unable to obtain details about how his data was being used or how it might affect his interactions with the police. His case underscores the broader ethical dilemma of AI in policing: how can individuals be held accountable for decisions made by algorithms they cannot understand or challenge?
The Future of AI in Policing
Despite the failures of the Avon and Somerset program, the UK government is pushing ahead with AI in law enforcement. The recently established PoliceAI, backed by £75 million in funding, aims to roll out AI tools to police forces across England and Wales. Andy Marsh, the former chief constable of Avon and Somerset and now the CEO of the College of Policing, has described AI as a 'heroin-like' solution for speeding up police work. However, the investigation raises serious questions about whether such tools can be deployed responsibly without addressing the underlying issues of transparency, bias, and public trust.
Key Takeaways
- Transparency is Essential: AI systems in policing must be transparent, with clear explanations of how data is used and how decisions are made.
- Bias Must Be Addressed: Predictive models must be rigorously tested for bias to ensure they do not disproportionately target marginalized communities.
- Public Engagement is Crucial: Law enforcement must engage with the public to build trust and ensure that AI tools are seen as legitimate.
Conclusion
The UK's experiment with predictive policing highlights both the potential and the pitfalls of AI in law enforcement. While AI has the potential to revolutionize policing, it must be deployed responsibly, with a focus on transparency, fairness, and public trust. Failure to address these issues could undermine the very goals that AI is intended to achieve: making communities safer and more just.