Police Avon and Somerset and Bristol City Council stopped using at least two AI models that assessed the risk of crimes against children. The reason was poor accuracy, and the systems were almost impossible to audit—independent reviewers could not find the source code or a list of variables, WIRED reported.
The outlet, together with rights group Liberty Investigates, local outlet Bristol Cable, and nonprofit newsroom Lighthouse Reports, analyzed hundreds of pages of documents obtained via freedom-of-information requests. The report comes as PoliceAI, a national center for testing and scaling AI tools across police forces in England and Wales, gets underway.
How Bristol gathered data
The effort centered on the Think Family Database, a Bristol City Council database launched in 2016 to support work with families and children who might need assistance. According to WIRED, it could have included records on nearly 500,000 city residents, although the project page now refers to about 55,000 families. The discrepancy may reflect the difference between individual records and family profiles.
The Think Family Database combined police and social data: housing status, information on mental health, teenage pregnancies, participation in parenting courses, school absences, and receipt of free school meals. The outlet said the information was collected without residents’ explicit consent, relying on legal bases for data sharing among government bodies.
One police data specialist described the approach as mixing disparate datasets.
“I put it all into a big bucket,” he said.
Using the Think Family Database, law enforcement and authorities built machine learning models that assigned risk scores to adults and children. Journalists identified at least 23 models at Avon and Somerset Police—ranging from predicting burglaries and failures to appear in court to the risk of a person going missing and the likelihood of becoming a victim of domestic violence.
In parallel, an Offender Management App covered around 300,000 people in the region. A senior officer described it as the basis for a “leaderboard” of the most dangerous offenders.
Why the models were shut down
One early model assessed the risk of crimes against children. According to WIRED, it combined data from police, the city council, and other government bodies, as well as anonymized information from the charity Barnardo’s on 1,000 children who had already been victims of such crimes.
The scoring also took into account:
- a child’s “in need” status;
- persistent school absences;
- mental health issues.
Another model considered housing support, rent arrears, and free school meals.
In 2016, the police ethics committee warned that the selected data and variables could lead to algorithmic bias. It recommended using the system with caution and explaining in advance to the public why and how such analytics would be applied.
The project was later reviewed by UK nonprofit consultancy Social Finance. The review called risk scoring the weakest element and said low accuracy undermined the models’ practical value. By the time of the assessment, two models for evaluating the risk of crimes against children had already been discontinued, WIRED wrote.
Social Finance linked the deterioration in model quality to changes in the dataset. Police tried to scale the approach across Avon and Somerset, which covers five local councils, but failed to reach data-sharing agreements with all local authorities. As a result, the models were left with a predominantly police “core” without the earlier social indicators.
According to the reporters, Bristol city services staff complained that vulnerable children were not appearing in the results. One employee wrote that minors who had recently become victims could receive a lower score than burglary suspects. Other staff said they were not prepared to rely on the scores due to the opaque methodology.
Social Finance also could not fully audit the models: the source code and list of variables could not be found. By June 2023, neither the police nor Bristol City Council had retained documents on the decision to drop the two child-crime risk models, WIRED reported.
What the audit showed
Separately, WIRED obtained from police more than 36,000 performance scores for 13 models that were used or tested between 2017 and 2024. The outlet shared the dataset with audit firm Eticas. The firm concluded that most models had low precision for positive hits—meaning the systems incorrectly flagged a significant share of people as at risk.
According to the data, for more than three years the model aimed at identifying potential burglars had positive precision below 10%: fewer than one in ten people flagged by the system actually committed such a crime. Auditors also noted that such figures are atypical for well-managed models in operational use.
Police told WIRED that some models, including the burglary tool, were not deployed. They attributed the presence of multi-year performance scores to automatic checks of a static file that was not removed after the decision not to deploy.
A page on the site says some of the force’s tools use AI and that algorithmic results serve only as advisory signals for staff. Police emphasized that the models do not make decisions automatically.
Bristol City Council said it currently uses only the NEET risk model—an assessment of the likelihood that a child will be not in education, employment or training after leaving school. The authorities said the tool does not replace professional judgment.
PoliceAI
The report comes amid the expansion of AI use in law enforcement via PoliceAI. On June 10, the UK Home Office launched a center to test and scale AI tools across 43 police forces in England and Wales. The project budget is £75 million over three years.
In its first year, PoliceAI will focus on tools for processing, disclosure, and summarizing digital evidence. Trials are expected to take place in ten forces in 2026–2027, followed by scaling across all police forces.
PoliceAI operates within the College of Policing—the professional body that sets standards and training for officers in England and Wales. It is led by former Avon and Somerset Police chief constable Andy Marsh.
WIRED noted the connection: the region that developed the disputed AI analytics previously employed the head of the body now involved in scaling AI for police. Against this backdrop, the Bristol case shows that the risks of such models relate not only to algorithmic accuracy but also to data quality, documentation retention, and the ability to conduct independent audits.
Earlier, the outlet reported on Maryland resident Alonzo Sawyer, who spent nine days in jail after a mistaken match in a facial recognition system.
Earlier, London Mayor Sadiq Khan blocked a contract worth nearly £50 million between Palantir’s UK unit and the Metropolitan Police. The deal envisioned implementing the AI-based Unified Operational Analytics system to speed up criminal investigations. The mayor vetoed the agreement, citing “serious breaches” of procurement procedures.
In 2023, it emerged that the U.S. Federal Bureau of Investigation, together with the Pentagon, had secretly developed and tested a facial recognition system.
