Human-on-the-Loop Runbook: Escalation Tiers When Agents Act
Escalation tiers T0–T3 for agentic workflows: when agents may read, draft, or act—and the runbooks, kill switches, and on-call patterns that keep autonomy bounded.
Executive Summary
By mapping all write-actions to four distinct tiers with hard spend caps, mean time to detect bad autonomy dropped from six hours to under 45 minutes. The critical shift was tying tier violations directly to an on-call P&L owner's pager, not a generic engineering queue, forcing accountability at the point of action.
"An un-tiered support agent that could auto-close tickets cost $75,000 in erroneously closed revenue accounts before a formal escalation framework was implemented."
The Challenge
The initial mandate was to increase support efficiency using a large language model (LLM) powered agent. The agent could draft replies, search the knowledge base, and—critically—close tickets that were inactive for 72 hours. The product team measured success by the percentage of tickets auto-resolved. The operations team was left to handle the consequences.
A composite scenario: An agent misclassified a high-value enterprise customer's negotiation inquiry as a routine support request. After 72 hours of perceived inactivity (the customer was conferring internally), the agent closed the ticket. The account, worth over $100k in annual recurring revenue (ARR), churned, citing unresponsiveness. The incident triggered a leadership demand for “more human review,” a directive too vague to be actionable. The core problem wasn't a lack of humans, but a lack of disciplined process for when and how they should intervene. The mean time to detect this failure was over six hours, discovered only after the customer’s complaint reached a VP. The stakes were clear: enable agent autonomy without creating unbounded financial and reputational risk.
The Approach
The question was how to move from ad hoc panic to a repeatable system of controls. We rejected a single, centralized human approval queue; it would become a bottleneck and dilute context. Instead, a cross-functional team of Product, Engineering, and Support Operations was tasked with designing a system of tiered, delegated authority. The approach was modeled on high-reliability organizational principles seen in aviation or medicine (Reason, 1997), where specific triggers require specific, pre-planned responses.
Our working group’s first principle was that risk is a function of the action, not the agent. We mapped every possible agent action—from summarizing a case to issuing a refund—onto a risk matrix. The trade-off was explicit: higher-risk actions would be subject to tighter constraints, slower review cycles, and more intrusive monitoring, reducing agent velocity in exchange for safety. This aligned with the core tenets of the NIST AI Risk Management Framework, focusing on governing and managing outcomes [1]. The output was not a new dashboard, but a binding contract: the Escalation Tier Framework. Ownership was assigned per workflow, making a specific Head of Support or Revenue Operations manager accountable for the actions taken within their domain.
Escalation Tier Framework
The framework classifies all agent actions into four tiers, balancing the volume of actions against their potential impact. Most operations should live in T0 and T1, with T2 and T3 reserved for workflows where the value of autonomous action justifies the cost of stricter controls.
Figure 3: Tier vs Agent may vs Human role
The framework classifies all agent actions into four tiers, balancing the volume of actions against their potential impact.
Tier: T0 – Read · Agent may: Retrieve, summarize, classify, tag internal-only data. · Human role: Spot-check samples for classification accuracy. · Example controls: No customer-visible writes. Read-only API keys.
Tier: T1 – Draft · Agent may: Propose text/code; stage changes in a sandbox; generate internal reports. · Human role: Approve batch or per item before execution. Review is mandatory. · Example controls: Queue + SLA for review (e.g., 4 hours). Changes are not live until approved.
Tier: T2 – Act (bounded) · Agent may: Execute allowlisted tools under spend/scope caps. Auto-close low-priority tickets. · Human role: Monitor anomaly dashboards; override flagged actions; investigate exceptions. · Example controls: Per-action spend cap (e.g., <$50 refund). Strict tool allowlist. Rate limits.
Tier: T3 – Act (elevated) · Agent may: Cross-system writes; process refunds; send external email to high-value cohorts. · Human role: Real-time gate, dual control, or break-glass approval for every action. · Example controls: Break-glass UI + mandatory audit log. Multi-person approval for actions >$500.
The Results
Implementing the tier framework and associated runbooks produced auditable improvements within one quarter. The primary goal—preventing high-stakes failures while enabling low-stakes automation—was met by focusing on detection time and accountability.
The metrics dashboard tells the story: Prior to the framework, mean time to detect (MTTD) a tier violation was effectively infinite, as tiers didn't exist; detection of the inciting incident took over six hours. After implementation, automated alerts on T2+ spend and escalation rates drove MTTD for write-actions down to an average of 38 minutes. All T2 and T3 actions achieved 100% trace bundle coverage, ending blame-focused post-mortems and enabling process-focused improvements. Most critically, the rule preventing auto-closure on high-value intents (Band C) was validated at zero incidents in the 90 days following rollout, directly addressing the failure that prompted the initiative.
Figure 4: Metric vs Target vs Result (90 days post-launch)
The metrics dashboard tells the story: Prior to the framework, mean time to detect (MTTD) a tier violation was effectively infinite, as tiers didn't exist; detection of the inciting incident took o…
Metric: Mean time to detect tier violation · Target: < 60 min · Result (90 days post-launch): 38 min
Metric: % T2+ actions with trace bundle · Target: 100% · Result (90 days post-launch): 100%
Metric: Escalations reviewed within 24h · Target: > 95% · Result (90 days post-launch): 98%
Metric: Auto-close without human on Band C intents · Target: 0 · Result (90 days post-launch): 0
Runbook Essentials
Pre-launch Checklist:
[ ] Action Inventory: Document every tool and write-action the agent can perform.
[ ] Tier Mapping Workshop: For each action, assign a tier with the workflow owner (Product, Ops, Eng).
[ ] Control Implementation: Configure spend caps, rate limits, and tool allowlists in production.
[ ] Kill Switch Test: Document and test the process for disabling T2+ tools. A real engineer must run the test and confirm it works in under 5 minutes.
[ ] On-Call Assignment: Assign a primary and secondary on-call owner for each T2+ workflow. Publish the roster.
Release week: Eval gate on T2+ tools; shadow mode for new tools 48h; daily review of escalations triggered.
Incident: Freeze tool allowlist; roll back prompt/version; preserve trace bundle (prompt, tools, outputs) for post-mortem.
Trace bundle (minimum)
Every T2+ action log must include: workflow_id, tier, prompt_version, model_id, tool_calls[], human_override boolean, and spend_usd estimate. Post-mortems without a trace bundle are closed as “insufficient data”—no blame assignment. This is a non-negotiable requirement for any team wishing to ship an autonomous agent.
Escalation playbook (paste-ready)
Detect — Anomaly on escalation rate or spend cap (pager).
Contain — Disable write tools for workflow via operational panel; leave read paths on.
Diagnose — Compare trace bundle to last known good release. The on-call owner is responsible for the initial diagnosis.
Recover — Roll back prompt/model; replay failed actions from queue with human review.
Learn — Add failure to eval golden set if revenue path; adjust tier map if policy gap.
Implementation (30 / 60 / 90)
30 days: The focus is on visibility and establishing a baseline.
Tier map for the top 3 revenue-impactful workflows (e.g., support, sales ops, finance).
Implement structured logging for all T2+ actions. The trace bundle must be available in your logging tool.
Publish the on-call card for these workflows and conduct a tabletop exercise of the escalation playbook.
60 days: The focus is on automated enforcement and training.
Wire T2+ tier violation alerts (spend cap exceeded, disallowed tool called) directly to the on-call owner's pager.
Implement hard caps on auto-close and refund behaviors based on customer segmentation.
Train all support leads and ops managers on how to use the override UI and where to find trace bundles. Run a fire drill of a live (but non-critical) incident.
90 days: The focus is on governance and continuous improvement.
Institute a mandatory quarterly tier reclassification review for all T2+ workflows.
Tie the estimated cost of T2+ agent actions to the owning department’s budget (chargeback).
Formally retire any agent tools that repeatedly fail tier policy tests or cause frequent escalations.
The Analysis
Figure 1: Escalation tier volume vs risk
Most traffic stays T0–T1; T2–T3 reserved for highest-stakes actions.The chart above quantifies the core trade-off for human on the loop runbook.
Most traffic stays T0–T1; T2–T3 reserved for highest-stakes actions.Most traffic stays T0–T1; T2–T3 reserved for highest-stakes actions.Tier maps fail when write tools ship without spend caps. The most common failure pattern we observed early on was treating the framework as documentation rather than implementation. A Tier 2 (bounded act) assignment must be coupled with programmatic enforcement—per-action dollar ceilings and tool allowlists tested in staging. Trace bundles are the foundation of learning; without the prompt_version, tool_call array, and spend_usd estimate, any incident analysis is just speculation. The metric that mattered, mean time to detect, only dropped below sixty minutes when escalation-rate alerts paged workflow owners directly, not a generic platform rotation.
Rubber-stamp approval defeats Tier 1 (draft) design. Early on, we saw managers batch-approving dozens of agent-drafted emails without review. The fix was in the user experience: for high-stakes customer segments, the UI was changed to require per-item review and force a "reason for approval" comment.
Finally, kill switches that require a full deployment are ceremonial safety theater. They are too slow to be effective. Product teams must provide operational levers—rate limit toggles, spend cap dials, tool disable buttons—that the on-call operator can use without filing a ticket with engineering. The quarterly tier reclassification forces a conversation about whether a workflow’s autonomy is earned, preventing "trust creep" where a T1 tool slowly evolves T2 capabilities without review.
What Went Wrong
Our first kill switch was a feature flag in the deployment pipeline. During a minor incident where an agent began sending misformatted emails, it took 45 minutes for an on-call engineer to get access, run the build, and deploy the change that disabled the tool. This was far too slow. The agent sent several hundred incorrect emails before it was contained. The lesson was that operational controls must be decoupled from the core release cycle. We rebuilt the kill switches into an internal admin panel accessible by the on-call support lead, cutting containment time to under two minutes. Operational readiness is not a feature flag; it's a dedicated, tested, operator-controlled interface.
Canonical scope
Archive note (June 2026): Public canonical for the agent autonomy hitl cluster. Operator T0–T3 escalation runbook for agentic workflows. Sibling case studies remain in the editorial backlog until differentiated.
Methodology & limitations
This analysis uses composite operator scenarios and illustrative chart values for teaching—not a single client outcome study. Adjust for your domain before production decisions.
References
Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://www.nist.gov/itl/ai-risk-management-framework
Monday Morning Checklist
[ ] Assign a named P&L owner for
human-on-the-loop-runbookin the Decision Ledger.[ ] Export top 10 inference paths by spend (last 30 days) with
workflow_idtags.[ ] Document three kill-switch triggers: spend ceiling ($/hour), error rate (%), human-escalation rate (%).
[ ] Stand up pre/post validators on one customer-data workflow; define three fail-closed reason codes.
[ ] Align model risk / compliance on bundle versioning for prompt + retrieval + rules.
[ ] Schedule 30-minute review with finance: walk Figure 1 ledger for human on the loop runbook; agree showback vs. chargeback date.
Editorial transparency. Essays at The AI Operator may use AI-assisted research, drafting, and editing tools under staff editorial review. Facts, figures, and recommendations are checked before publication; we correct the record when evidence changes. Questions: hello@theaioperator.net.
Published on [Substack](https://theaioperator2.substack.com/p/human-on-the-loop-runbook).