What Happened
St. Luke's University Health Network runs incident response operations where speed is critical. Like many security teams, they've explored AI-driven automation to manage the workload. However, associate CISO Krista Arndt identified a risk pattern before it became a problem: AI systems making decisions on critical infrastructure without human oversight.
The risk: an AI agent closes what it considers low-severity alerts, but these alerts share a common thread only visible when aggregated. By the time a human analyst notices the pattern, an attacker could have moved laterally through systems the AI labeled "low-risk."
This isn't a breach analysis. It's a near-miss evaluation, and it's more insightful than most incident reports because Arndt identified the structural weakness before it led to a crisis.
Timeline of the Risk Pattern
Week 1-2: The security team evaluates AI tools promising to automate tier-1 SOC tasks. Vendor demos highlight impressive ticket closure rates and mean-time-to-resolution improvements.
Week 3: The team considers autonomous remediation for alerts the AI scores below a certain threshold. The business case is strong because analyst time is costly and alert volume is outpacing headcount.
Week 4: Arndt's team maps out which systems the AI would affect. They realize that "low-risk" alerts on authentication systems, if closed without correlation, could hide a credential stuffing campaign. A single alert about failed logins on a development server isn't critical. Fifty similar alerts across dev, staging, and internal tools over three days indicate reconnaissance.
Decision point: Instead of autonomous closure, they implement AI-assisted triage with mandatory human approval for any action affecting identity systems, critical infrastructure, or involving pattern recognition across multiple alerts.
Which Controls Failed or Were Missing
The controls didn't fail because Arndt's team established the right governance framework before deployment. However, the incident-that-didn't-happen reveals potential failures:
Lack of action-level risk classification. The AI vendor's "low-risk" category grouped authentication failures with certificate renewal notices. One is operationally benign; the other could be an Indicator of Compromise (IoC) if it repeats.
No correlation requirement before closure. The AI evaluated each alert in isolation without checking for similar alerts in the queue or closed in the previous 72 hours.
Missing human-in-the-loop checkpoints. The proposed workflow let the AI close tickets without logging which analyst would have handled it manually or creating an audit trail for human review.
Insufficient rollback capability. If the AI closed 30 alerts and an analyst later identified them as related, there was no easy way to reopen all of them and escalate the cluster as a single incident.
What the Relevant Standards Require
No PCI DSS requirement explicitly states "don't let AI close security alerts without oversight," but several requirements apply when you map the workflow:
PCI DSS 10.6 requires log review. If your AI closes alerts generated from logs, you're still responsible for ensuring someone reviews the closure decisions. Automated closure without audit trails doesn't meet the "review" standard.
PCI DSS 6.3.2 (in version 4.0) requires that security testing and monitoring processes remain effective when you change them. Replacing human triage with AI triage is a change. You need to validate that the new process catches what the old process did.
NIST Cybersecurity Framework (CSF) DE.AE-3 calls for correlating event data from multiple sources. If your AI closes alerts without checking for correlation, you're not meeting this detection objective.
NIST SP 800-53 IR-4 covers incident handling. It doesn't prohibit automation, but it requires that your incident response process includes analysis, containment, and lessons learned. If the AI's action log doesn't support retrospective analysis, you can't demonstrate compliance.
The issue isn't that AI violates these standards. It's that autonomous AI can make it impossible to prove you've met them.
Lessons and Action Items for Your Team
Arndt's guidance translates into five operational rules you can implement this quarter:
1. Map every AI action to a reversibility window. If the AI closes an alert, you should be able to reopen it and restore the original context within 24 hours. If it blocks an IP address, you need a rollback procedure that doesn't require a change request. Build the undo button before deploying automation.
2. Require human approval for any action on identity or access systems. Your authentication logs, Multi-Factor Authentication (MFA) events, and Role-Based Access Control (RBAC) changes are too sensitive for autonomous closure. Even if the AI is 99% accurate, the 1% error rate on identity alerts can let an attacker establish persistence.
3. Implement correlation checks before closure. Your AI should query: "Are there other alerts in the past 72 hours with the same source IP, user account, or affected system?" If yes, escalate to a human. If no, log the closure with enough detail that an analyst reviewing the week's activity can spot patterns the AI missed.
4. Audit AI decisions weekly, not quarterly. Assign one analyst to review a sample of AI-closed tickets every Friday. You're looking for false negatives (alerts the AI dismissed that should have escalated) and pattern gaps (alerts that look unrelated individually but form a sequence).
5. Strengthen foundational controls before adding AI layers. Arndt's point about technical debt applies here. If your asset inventory is incomplete, your AI will make bad decisions about which systems are "low-risk." If your data loss prevention rules haven't been updated in two years, the AI will inherit those gaps. Fix your identity management, asset tracking, and baseline monitoring before you automate on top of them.
Your AI vendor will show you metrics about efficiency gains. Ask them to show you the audit trail for a false negative. If they can't produce one, you're buying speed at the cost of visibility.
Action checklist:
- Document which alert types require human approval regardless of AI confidence score
- Build a weekly AI decision audit into your SOC workflow
- Test your rollback procedure: can you reopen and re-investigate an AI-closed alert in under 30 minutes?
- Review your asset inventory and identity logs, are they accurate enough to support automated risk scoring?
- Create an escalation rule: if the AI sees three similar alerts in 72 hours, route them to a human analyst as a cluster
