Skip to content
AI · SOC JUL 2026 · 6 MIN read

Auditing the triage agent: 94% noise, 0 lost cases

The reasoning log behind every dismissal, the monthly blind sampling and how a client can review it.

AI
R&D · SOC AI
CROC agentic operation
Service: CROC 24/7 →
THE ESSENTIALS

The agent doesn't decide what is an incident: it decides what deserves a person's attention, and logs why it dismissed the rest.

Every dismissal is auditable and reversible; the analyst can pull the full reasoning chain of any alert.

The metric that matters is not the percentage dismissed but how many real cases survived the filter. In twelve months: all of them.

The conversation about AI in security operations usually starts on the wrong side: how much can be automated. The right question is which part of the work consumes analyst time without adding judgment. That part is volume triage.

The problem isn’t volume, it’s repetition

A shift receives thousands of alerts. The vast majority resolve with the same sequence: who is the user, is the behavior usual, was the destination already known, is anything else happening around it. An experienced analyst does that in ninety seconds and concludes the same thing 94% of the time: it’s nothing. Ninety seconds times thousands is the whole shift, and the real case arrives exactly when the analyst is exhausted from dismissing.

The dismissal log

The piece that makes the design acceptable is not the model, it’s the log. Every dismissed alert stores the question the agent asked itself, the evidence it consulted, the conclusion and the declared confidence. Any dismissal from the month can be opened and reviewed in full.

DISMISSAL LOG · ABBREVIATED EXAMPLE

alert       : Outbound connection to newly registered domain
hypothesis  : Legitimate software update?
evidence    : signed process · destination known for 41 days ·
              timing matches the usual window
conclusion  : dismissed, routine update
confidence  : 0.96   reviewable: yes

HOW WE MEASURE THAT NOTHING WAS LOST

Every month an analyst reviews a blind sample of dismissals without seeing the agent’s conclusion. If a false dismissal appears, the reasoning rule is corrected and the affected window reprocessed. The same quality control applied to a junior analyst.

Automation you cannot audit is not automation: it’s a black box we entrusted the shift to.

The analyst stopped doing volume triage and went back to what only a person does: deciding with incomplete information, talking to the system owner, and answering for a containment that interrupts the business. The agent gave the time back; the judgment remains human.

Material like this, every week.

The CERT-CBRT bulletin: what is being exploited in the region and what the shift learns by operating. No marketing.