Alert Triage With AI: How Analyst-in-the-Loop Works Day to Day


Security teams in Korea are seeing the same pattern from different directions. The number of alerts grows with every new log source and every AI-assisted campaign, while the number of analysts on a night shift stays roughly the same. Jay Kim, threat analyst at the PAGO DeepACT MDR Center, described the problem at PAGO Security Summit 2026 in simple terms: attacks now scale to hundreds or thousands of targets at once, but an analyst's time does not scale with them.[2] Hiring alone cannot close that gap, which is why alert triage has become the first place most SOCs put AI to work.
Using AI for alert triage does not mean handing the decision to a model. In the operating model we run at PAGO, the AI investigates and assembles the evidence, and an analyst stays in the loop to review that evidence and decide. This article walks through what that looks like across an ordinary day and night, and how to tell whether it is working.
What AI alert triage actually does
Jay Kim breaks the work of an AI investigator into three steps. It operates within a defined scope, meaning the repetitive tasks and investigation boundaries it has been approved to handle. It chains context together by correlating alert, endpoint, identity, network and threat intelligence data that would otherwise sit in separate consoles. And it produces a verdict, a structured conclusion with the evidence that supports it, which is passed to an analyst for review.[2] Underneath the verdict sits an evidence pipeline that collects observations, enriches them with threat intelligence and GeoIP data, validates the evidence and then summarizes it.
In practice, this means an analyst receives one explained scenario instead of four unrelated alerts. An example from the session links an RDP login from an untrusted external source, PowerShell launched from that remote session, an encoded command that downloads a suspicious payload and the persistence attempt that follows, and classifies the whole chain as a true positive with evidence attached to each step.[2]

An evidence-based verdict from Jay Kim's session at PAGO Security Summit 2026]
The 3:04 a.m. test: where false positives and missed attacks come from
Consider an alert raised at 3:04 a.m. for suspicious PowerShell execution on a finance team workstation. An AI that can see only the endpoint alert may reasonably conclude that it is an administrator's script and close it. An AI that can also read identity and DNS logs from the same period notices that the same user signed in from an unusual country forty minutes earlier, and escalates the alert instead.
Both systems are using AI, and both are following their own logic correctly. What separates a missed attack from a correct escalation is the range of data the AI can see, together with an analyst who knows which questions to ask of it. In this scenario, the night-shift analyst reviews the evidence, agrees with the escalation and isolates the workstation. The next morning, the team reviews every verdict the AI reached overnight, and any decision an analyst overturned becomes a tuning item.
Who does what in the loop
Step | AI | Analyst |
Collect | Gathers related alerts, entities and behavior | – |
Enrich | Adds threat intelligence, GeoIP and history | – |
Conclude | Proposes a verdict with evidence and counter-evidence | Reviews the evidence |
Act | Prepares the response | Approves and executes |
Learn | Is retuned from corrections | Records why a verdict was wrong |
The split is deliberate. Collection and enrichment are high in volume, repetitive and easy to audit, so they belong to the AI. The decision to act, and the accountability for it, stay with a person. Jay Kim's framework describes it the same way: agentic AI carries out the investigation and evidence work, while final judgment and responsibility remain with people.[2]

Human in the loop in the DeepACT DEFENSE framework, from Jay Kim's session
Measuring the AI like a new hire
In his opening keynote at the same summit, Paul Kwon, CEO of PAGO Networks, argued that AI adoption in security should be handled like hiring a new team member, with a job description, guidelines, measurable performance targets and clear grounds for dismissal.[3] The keynote proposed five areas for evaluating that hire. Here they are with the example targets PAGO uses:
Area | Example target |
Efficiency | Analysis time from 2 hours to 5 minutes |
Error rate | Alert triage errors from 20% to 5% |
Detection | Mean time to detect from 2 hours to 30 minutes |
Skills | Junior analysts able to take on senior-level work |
Coverage | Share of telemetry that can be investigated from about 80% to 100% |
Two of the five areas, skills and coverage, describe what happens to the team rather than to the tool, which is a useful reminder that the goal of AI alert triage is a stronger SOC rather than a smaller one.
Why the analyst stays in the loop
Practitioners are clear about where they want accountability to sit. In a survey presented at the Gartner Security & Risk Management Summit 2026, 64% of respondents said AI output must come with human verification and accountability.[1] The loop also makes the AI better over time. When experienced analysts correct a verdict, the correction feeds back into tuning, and the AI in turn brings a consistency that is difficult to achieve across shifts, investigating the same type of alert the same way every time.
Jay Kim also pointed to the Jevons paradox in his session: as AI makes triage cheaper, the time it frees does not disappear but moves to higher-value work such as threat hunting and detection engineering.[2]
Three metrics for running the loop
Time to triage: from alert to analyst decision, reported separately for day and night shifts.
Override rate: how often analysts overturn the AI's verdict, and the reasons they record.
Feedback to fix: how long it takes for an analyst's correction to become a tuning change.
Before evaluating any AI triage product, it is worth mapping the current triage process step by step, from the moment an alert fires to the moment someone decides. That baseline is the only reliable way to see which steps the AI actually shortens.
Alert triage that improves with every decision
Keeping an analyst in the loop is not a slower form of automation. It is how the AI improves over time and how the organization keeps a named person accountable for each response. The AI absorbs the volume, analysts make the decisions, and each reviewed decision makes the next round of triage faster and more accurate.
Watch and read more
Watch the full session: "Accelerating Alert Triage with AI Investigator", presented by Jay Kim, Threat Analyst, PAGO DeepACT MDR Center, at PAGO Security Summit 2026.
Read the full report: PAGO Security Summit 2026 Report, free to read with no sign-up required.
References
Gartner Security & Risk Management Summit 2026 practitioner survey, as documented in the PAGO Security Summit 2026 Insight Report.
Jay Kim, "Accelerating Alert Triage with AI Investigator", PAGO Security Summit 2026.
Paul Kwon, "The Next Era of Security Operations: Beyond Detection, Toward Better Decisions", opening keynote, PAGO Security Summit 2026.



