From Hallucinations to Reality: A Critical Examination of LLM-Based Alert Investigation Tools

As security community rushes to adopt AI-powered security tools, Large Language Models (LLMs) are being marketed as a silver bullet for automating alert investigation. But can we trust LLMs to reliably investigate security incidents involving sensitive research data and student information? This session presents results that shows how known limitations of LLM-based investigation tools can negatively impact investigation, including hallucinated attack paths, inconsistent results, and compliance gaps that make them particularly unsuitable for higher education environments. Drawing work we have done at University of Illinois, we'll demonstrate these limitations through live examples and share detailed benchmarks showing why LLMs fall short of education sector requirements. More importantly, we'll introduce an alternative approach that achieves what LLMs promise - automated, reliable alert investigation that reduces analysis time from 30 minutes to 3 minutes all within existing infrastructure. Whether you're evaluating new security tools or looking to improve SOC efficiency, this session provides actionable insights for making informed decisions about emerging security technologies. You'll leave with clear evaluation criteria for assessing AI-powered tools and practical strategies for implementing more reliable alternatives that better serve higher education's unique needs.