The Eval That Pays Rent: When Production Regression Suites Beat Leaderboard Faith

A pricing copilot team shipped a model upgrade that passed every offline benchmark—and watched conversion on a $2.1M ARR workflow fall within 48 hours. The release gate was green on academic tasks; it was blind to revenue-path failures. This case memo documents how a 14-person platform org rebuilt eval around production failures with dollar tags, wired a business-weighted golden set into CI as a promote/hold gate, and raised eval failure → incident correlation from 0.12 to 0.61 within 90 days—while accepting higher eval infra spend because finance could see avoided incident cost.

This is narrow eval economics in production—not the full AI engineering discipline, not agentic security, not Operator Trilogy prompting. If your eval budget cannot explain a support ticket or SLA credit within 72 hours of a failure, it is R&D theater.