Key figure: The global average cost of a data breach was $4.44 million in 2025 (IBM, Cost of a Data Breach Report 2025).
Every cybersecurity news breach postmortem has the same opening line: at 3AM, the on-call engineer noticed something wrong. This drill puts you in that chair. Five escalating incidents, a 15-minute countdown, and an agent fleet doing things nobody approved. Incident response is a muscle — this is the gym.
Because attackers work when defenders sleep: off-hours mean slower detection, fewer responders, and automated systems making decisions alone. The 3AM cliché persists because dwell time — the gap between compromise and discovery — is still measured in days for the average breach. Drills like this one exist to shrink that gap: teams that have rehearsed the first fifteen minutes respond in minutes instead of hours when the real page comes.
Incident response is the organized process of detecting, containing, eradicating, and recovering from a security incident — plus the postmortem that prevents recurrence. The first hour, when evidence is fresh and damage is spreading, matters most.
Your phone just went off. Something in your AI agent fleet is wrong, and it gets weirder every few minutes. You have 15 minutes and five escalating stages. Each stage gives you three runbook actions — one is the right call, two look reasonable and will cost you. Choose like your job depends on it, because at 3AM it kind of does.
Five stages. Three runbook choices each. Fifteen minutes on the clock. Beat the pager.
Agent Auth Gateway is the runbook, enforced: it freezes rogue agents, revokes credentials, blocks unknown egress, and keeps tamper-evident logs — automatically, at 3AM, while you're still asleep.
Our platforms are being upgraded and return November 1 — get early access now, or explore the full Ingenious AI security suite:
Get Agent Auth Gateway early access →Trace every session per agent: which tools were called and with what arguments, token consumption and cost, latency, error and retry rates, and policy decisions. Alert when a rolling metric crosses a threshold — cost per session spiking, hallucination rates rising — and feed every failure back into your test suite so production reality keeps it honest.
Pages for AI-specific failure modes alongside your infrastructure runbooks: how to pause the agent, what to audit, how to assess downstream impact, and who decides on rollback or re-enablement. Write them for a 3AM brain, keep them next to the code, and test them in exercises before you need them — an untested runbook is a plan, not a capability.
Contain, then investigate. Isolate the agent and disable its workflows, revoke its tokens and rotate secrets, and stop ongoing harm before you touch root cause. Triage severity by two questions: what can this agent reach, and what has it already done — the trace history answers the second one directly.
Watch the baselines: tool-call frequency spiking far above normal, permission-denial spikes for operations the agent never attempted before, new or unusual outbound destinations, output sizes swinging wildly, and — the big red one — an agent tampering with its own logging. Treat log tampering as a compromise until proven otherwise.
More drills, autopsies, and war stories from the agent-security lab:
A fictional interactive drill: five escalating AI agent incidents, a 15-minute countdown, and three runbook choices per stage — scored, with the right call explained for every stage.
Trace every session per agent — tool calls, tokens and cost, latency, errors, policy decisions — alert when a rolling metric crosses a threshold, and feed every failure back into your test suite.
Contain, then investigate. Isolate the agent, revoke its credentials, and stop ongoing harm before you touch root cause.
Tool-call frequency spikes, permission-denial spikes, new or unusual outbound destinations, output-size anomalies, and an agent tampering with its own logging — treat log tampering as a compromise until proven otherwise.