Students across the United States are increasingly being summoned to school offices or even arrested after AI-powered surveillance software flags their online chats as potential threats — often based on false alarms.
Lesley Mathis’s 13-year-old daughter learned this the hard way last year. The Tennessee eighth grader made an offensive joke during an online chat with classmates, triggering her school’s monitoring system. Within hours, she was arrested, interrogated, strip-searched, and detained overnight in jail, her mother said.
The incident began when friends teased the girl about her tanned skin, calling her “Mexican,” though she is not. In reply, she wrote, “on Thursday we kill all the Mexico’s.” Mathis admitted the comment was “wrong” but stressed the context showed no real threat.
“It made me feel like, is this the America we live in?” Mathis said. “It was this stupid technology that picks up random words without understanding context.”
Schools across the US use AI-based surveillance tools such as Gaggle and Lightspeed Alert to monitor students’ online activity on school devices and accounts, aiming to detect signs of self-harm, bullying, or violence. While educators praise the technology for saving lives, critics warn it often criminalizes innocent remarks.
New report reveals ChatGPT’s alarming responses to teens on sensitive issues
Elizabeth Laird, director at the Center for Democracy and Technology, said, “It has routinized law enforcement access and presence in students’ lives, including at home.”
In Tennessee, a 2023 zero-tolerance law requires any threat of mass violence against schools to be reported to law enforcement immediately. The 13-year-old girl’s arrest followed this policy, though Gaggle’s CEO Jeff Patterson criticized the handling, saying the system is meant to intervene early, not involve police.
Private student chats are also monitored. In Florida, a teenager was arrested after Snapchat’s automated system flagged a joke about school shootings. Alexa Manganiotis, a student in West Palm Beach, noted how quickly surveillance software works, sharing that students typing threats on school computers were removed within minutes.
Amy Bennett, Lightspeed Systems’ chief of staff, said the software helps schools “be proactive rather than punitive.”
However, data reveals high false alarm rates. An analysis of Gaggle alerts in Kansas found nearly two-thirds were nonissues, including false positives from homework or deleted photos.
Natasha Torkzaban, a recent graduate, was flagged for editing a friend’s college essay containing the phrase “mental health.” She and other students recently sued their school district, alleging unconstitutional surveillance.
School officials argue the technology has prevented dozens of imminent suicide or violence threats. “Sometimes you have to look at the trade for the greater good,” said Anne Costello, a Kansas school board member.
Two years after the incident, Mathis said her daughter is recovering but remains “terrified” of encountering the officers involved. She praised teachers at her daughter’s alternative school for their compassion and understanding.
“We just want kids to be these little soldiers, but they’re not,” Mathis said. “They’re just humans.”