Tackling Alert Overload with ChatOps-Integrated Alert Handling and Incident Response

Higher-ed Security Operations teams have to deal with high volume alerts and incidents emerging from a complex and dynamic academic computing environment, with limited staffing and budget. Sure, every vendor would be happy to sell you a pre-packaged all-in-one solution--it'll just cost you hundreds of thousands of dollars per year and require dedicated staff to keep it working. That's never been our way--we decided to pull together our experience with IT operations and integration/automation to build something custom fit to our needs and budget. In doing so, we discovered that modern Security Operations has a lot of parallels and can share a lot of tooling with modern IT Operations. We present a continuously evolving project: our alert management, incident response, and ChatOps automation system. We ingest alerts from numerous systems; standardize their format, automatically associate assets, users, and departments with alerts; de-duplicate them; identify which alerts can be handled automatically and which require human attention; and generate interactive chat notifications. Alerts found to be actual incidents are fed into an IT incident management tool for managing the incident response process (customized to our organization) and tracked through to resolution, with an emphasis on retrospective and metrics collection.