The Routing Stack: When Cascade Models Beat One-Size-Fits-All Inference
One flagship model for every call overspends on trivial requests—a tiered routing stack cuts inference 30–45% when cascades are observable and tied to $/decision.
Executive Summary
Most production teams route every large language model (LLM) call through one flagship model because it simplifies eval and avoids "wrong tier" incidents. That default is expensive. A composite workload—support triage, draft generation, and legal escalation—does not need the same inference profile on every request. By implementing a tiered routing stack, a composite operator reduced inference spend by 34%—saving over $16,000 per month—with no measurable drop in quality scores on its highest-volume workflows.
A routing stack beats one-size-fits-all when routing is observable and tied to $/decision. A small classifier or rules layer sends easy requests to a mid-tier model, escalates ambiguous cases to a flagship, and logs every branch. Finance sees which tier paid for which outcome—not a single blended rate hiding 40% overspend on trivial calls.
Figure 1: Routing cost by tier
Tiered routing reserves flagship spend for escalation paths.Scout → workhorse → expert tier economics.
This memo is about inference economics, not model selection theology. The Chargeback Imperative names who pays; here the question is which model tier should execute each request once ownership exists.
The Challenge
A composite mid-market SaaS operator ran 2.1 million LLM calls per month across three distinct workflows: customer support triage, internal memo drafting, and contract clause review. Initially, every call used the same flagship model at an composite operator rate of $0.018 per 1K output tokens. This single-model approach was chosen for its simplicity; one endpoint, one integration, and one quality evaluation benchmark simplified initial deployment and de-risked the launch.
The Head of Platform Engineering owned the model integration, while Product Managers for each workflow owned the user-facing features. The blended cost was simply absorbed into a central R&D budget, masking the underlying economics. The problem was that the workflows had vastly different requirements.
Figure 3: Tier vs Share of calls (before) vs $/decision (before)
The Head of Platform Engineering owned the model integration, while Product Managers for each workflow owned the user-facing features.
Tier: Flagship only · Share of calls (before): 100% · $/decision (before): $0.24 · Quality gate: Pass
Tier: Small classifier + mid-tier · Share of calls (before): — · $/decision (before): — · Quality gate: —
Tier: Mid-tier + flagship escalation · Share of calls (before): — · $/decision (before): — · Quality gate: —
Support triage, consuming 68% of total call volume, primarily required simple classification labels and short, formulaic replies. Contract review, on the other hand, demanded flagship-level nuance for complex legal language, but only on about 12% of requests after a cheap pre-screen for standard clauses. Running the flagship model on every support ticket was like using a specialist surgeon for every intake form—effective, but financially indefensible at scale. The Head of Finance flagged the escalating, undifferentiated LLM spend as a top-three R&D cost concern, forcing a review.
The Approach
The chart above quantifies the core trade-off for model routing stack.
The question became: how do we match the cost of inference to the value of the request without introducing massive engineering overhead or quality risk? The answer was a cascade routing system, owned by the Platform Engineering team as a centralized service.
Cascade routing has three layers, not two. Layer 1 is a cheap router—a set of rules, an embeddings similarity search, or a small classifier model—that assigns intent and a risk score. Layer 2 is a mid-tier model optimized for speed and cost-efficiency on draft-quality output. Layer 3 is the flagship model, reserved for escalations when the router's confidence falls below a set threshold, policy flags fire (e.g., mention of termination clauses in a contract), or human review is pre-committed.
The team’s first decision was the router mechanism itself.
Rules-based: A simple keyword and regex engine was prototyped for support triage. It was fast and transparent but failed on 30% of inputs with novel phrasing, incorrectly escalating simple requests.
Embedding-based: This offered semantic understanding but added significant latency (150-250ms) and cost to every call, eroding some of the savings.
Small Classifier: The team settled on a fine-tuned DistilBERT model for the triage and memo workflows. Trained on 20,000 historical requests, it provided a confidence score for each predicted intent with minimal latency (<50ms). For contract review, a simpler rules-based system was sufficient to identify standard clauses, escalating anything non-standard.
Thresholds were set against business risk, not just model accuracy. For support triage, the escalation threshold was set to a 90% confidence score. If the router was less than 90% sure of the user's intent, the request was sent to the flagship model. This meant accepting that some simple requests would be over-serviced to ensure no critical issues were mishandled by the mid-tier model. For memo drafting, the threshold was lower (75%), as the consequence of a poor draft was minimal.
Latency budgets belong in the router spec. The finance team was pleased with cost savings, but the support team initially resisted. The mid-tier path, despite a faster model, added a median 180ms of end-to-end latency due to the router's processing. To prevent teams from bypassing the stack, the platform group established and guaranteed tier-specific service level objectives (SLOs): support triage p95 under 2.4s, contract review p95 under 8s. Escalation to the flagship was allowed to break the triage SLO only when a policy flag fired—not when the router was merely uncertain. This constraint prevented "cheap but slow" routing that would have pushed support operators to find workarounds.
The Results
After deploying the routing stack and observing for 90 days, the team measured outcomes against the flagship-only baseline. The results were tracked across cost, quality, and performance.
Figure 4: Workflow vs Calls/mo vs Flagship share (after)
After deploying the routing stack and observing for 90 days, the team measured outcomes against the flagship-only baseline.
Workflow: Support triage · Calls/mo: 1,428,000 · Flagship share (after): 4% · $/decision (after): $0.09 · Δ spend: −38%
Workflow: Memo drafting · Calls/mo: 462,000 · Flagship share (after): 18% · $/decision (after): $0.14 · Δ spend: −31%
Workflow: Contract review · Calls/mo: 210,000 · Flagship share (after): 41% · $/decision (after): $0.31 · Δ spend: −12%
Composite savings reached 34% on inference spend with no measurable drop in human-reviewed quality scores for the triage and memo workflows. The key was moving high-volume, low-complexity tasks off the most expensive tier, not starving difficult cases of the powerful reasoning they required. The contract review workflow saw the smallest savings, but correctly concentrated flagship spend on the highest-risk legal analysis.
The router itself performed with 94% accuracy in classifying requests, ensuring most calls took the cost-effective path. Latency SLOs were met 99.8% of the time, securing adoption from the internal teams.
What Went Wrong
The initial rollout wasn't seamless. The primary failure mode was silent degradation in the memo-drafting workflow. In the first month, this workflow saw a 45% cost reduction, far exceeding the projected 31%. While Finance celebrated the variance, the Product Manager was alarmed. An investigation revealed the router's confidence threshold was set too aggressively based on offline tests. It was confidently routing complex, multi-topic draft requests to the mid-tier model.
The model didn't fail loudly; it produced grammatically correct but generic, vapid output that lacked the nuance of the flagship model. Users didn't file bug reports—they simply stopped using the feature. Active use dropped 20% in three weeks before the product analyst correlated low user engagement scores with high mid-tier routing rates. The system was "working" from a technical standpoint but failing from a user value perspective.
The fix required two changes. First, re-tuning the router threshold using online A/B testing data, not just the offline"
The fix required two changes. First, re-tuning the router threshold using online A/B testing data, not just the offline "golden set." Second, adding a simple "thumbs up/down" user feedback mechanism to the UI. This feedback now feeds directly into a weekly evaluation loop, allowing the platform team to catch model or router drift before it shows up as a drop in a lagging indicator like user retention.
What to Do Next
First 30 days: Baseline and Identify. Tag all existing LLM calls by their parent workflow. Compute the current, blended $/decision. Do not build anything yet. Use log analysis to identify the workflow where >50% of calls are short, structured, or classification-only. A simple proxy for complexity can be input/output token counts. This is your pilot candidate.
By 60 days: Pilot and Calibrate. Deploy a simple router on the pilot workflow only; keep the flagship model as the default for all other traffic. Set the initial escalation threshold conservatively (e.g., escalate if confidence is below 95%). Use production failure exports and user feedback—not offline accuracy alone—to tune this threshold down to a cost-effective but safe level.
By 90 days: Expand and Govern. Expand routing to a second workflow. Begin publishing a monthly report showing tier mix, $/decision, and quality scores to all workflow owners. Review escalation patterns weekly. If a workflow's flagship share climbs more than 5% above plan, it signals a problem. Fix the router or update the training data before cutting off access to tiers.
Operator Checklist: Router Readiness
[ ] Are workflows tagged with unique, machine-readable identifiers?
[ ] Is cost per workflow calculable (chargeback or showback in place)?
[ ] Does a "golden set" for regression testing quality exist for each workflow?
[ ] Are end-to-end latency SLOs defined and monitored?
Canonical scope
Archive note (June 2026): Public canonical for the routing finops cluster. Horizontal cascade architecture, telemetry, and $/decision. Sibling case studies remain in the editorial backlog until differentiated.
Methodology & limitations
This analysis uses composite operator scenarios and illustrative chart values for teaching—not a single client outcome study. Adjust for your domain before production decisions.
References
Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://www.nist.gov/itl/ai-risk-management-framework
Monday Morning Checklist
[ ] Assign a named P&L owner for
model-routing-stackin the Decision Ledger.[ ] Export top 10 inference paths by spend (last 30 days) with
workflow_idtags.[ ] Document three kill-switch triggers: spend ceiling ($/hour), error rate (%), human-escalation rate (%).
[ ] Pilot fail-closed validators on one high-risk path before expanding agent autonomy.
[ ] Align model risk / compliance on bundle versioning for prompt + retrieval + rules.
[ ] Schedule 30-minute review with finance: walk Figure 1 ledger for model routing stack; agree showback vs. chargeback date.
Editorial transparency. Essays at The AI Operator may use AI-assisted research, drafting, and editing tools under staff editorial review. Facts, figures, and recommendations are checked before publication; we correct the record when evidence changes. Questions: hello@theaioperator.net.
Published on [Substack](https://theaioperator2.substack.com/p/model-routing-stack).