Everybody using AI should know that it's prone to mistakes every once in a while, yet one major error became very costly for a Baltimore teen after they were left in handcuffs and surrounded by armed police.
One of the key features of artificial intelligence in recent years has been the absurd hallucinations that keep cropping up, no matter how hard the developers behind the tools try to stamp them out.
These can often be 'harmless' in a sense, like ChatGPT failing to follow the simple rules of a game of tic-tac-toe, yet they can be equally as dangerous, and could even lead to a loss in life in extreme scenarios.
That hasn't stopped law enforcement from taking advance of the tech as it continues to evolve though, and it appears as if a number of schools across the United States are using the software to identify 'threats'.
Advert
As reported by the BBC, 16-year-old Taki Allen had just finished football practice and had treated himself to a bad of Doritos, having put the empty packet in his pocked as he waited.
What soon greeted him though was a convoy of police cars filled with armed officers, as he was forced onto the ground with guns pointed at him as they believed he was in possession of a weapon.
School shootings are undeniably a major problem in America, with an alarming number having already occurred this year alone, and some schools have taken to experimental measures in order to prevent them before they occur.
Seemingly most popular among these are AI-based weapons detection systems, which review camera feeds to supposedly identify threats that might crop up within 'seconds'.
Allen was mistakenly identified by the Omnilert system employed by his school, and while human reviewers found no threat when they were alerted by the tech, the school's principal had missed this, contacting the safety team who in turn got in touch with the police.
"Police showed up, like eight cop cars," Taki Allen recalled, "and then they all came out with guns pointed at me talking about getting on the ground." The 16-year-old was also told to get on his knees before being arrested and put in handcuffs.
The Baltimore County Police Department has since insisted that its officers "responded appropriately and proportionally based on the information provided at the time," adding that "the incident was safely resolved after it was determined there was no threat."
However, one local politician has criticized the procedures that led to Allen's incorrect arrest, with local councilman Izzy Patoka writing on Facebook: "I am calling on Baltimore County Public Schools to review procedures around its AI-powered weapon detection system."
Omnilert, who provided the technology to Allen's school, assert that the chain of events that saw the student arrested and in handcuffs had "operated as designed," illustrating that "while the object was later determined not to be a firearm, the process functioned as intended: to prioritise safety and awareness through rapid human verification."
This has had an understandable and material affect on Allen's life at school though, as he now waits inside after football practice, claiming it's not "safe enough to go outside, especially eating a bag of chips or drinking something."
The student also added that "I don't think no chip bag should be mistaken for a gun at all."