Context Engineering for Operators: Budget, Grounding, and Refusal
Context budget, grounding policy, and structured refusal—three operator policies that beat bigger windows for legal and enterprise assist workflows.
Executive Summary
A legal-tech firm reduced erroneous AI-generated advice requiring human correction by 90% and cut its core workflow API costs by 62% in six weeks—not by upgrading its LLM, but by implementing a strict context budget that capped input tokens at 40% of the window's capacity. The team had been stuffing up to 40 kilobytes of tangential case files and emails into prompts, assuming more data meant better answers, which led to a 71% human approval rate and spiraling costs. The shift from maximizing context fill to maximizing context relevance treated the window as a constrained production asset, not an infinite scratchpad.
The Challenge
This memo uses a composite scenario from our work with mid-market legal and compliance technology firms. A product team at "LexiPro," a provider of AI-powered paralegal assistance tools, faced a critical performance bottleneck. Their flagship feature, which summarized case law and cross-referenced it with internal corporate policies, was failing in subtle but dangerous ways.
The key personnel were:
The Product Manager (PM): Tasked with increasing user adoption and satisfaction, she was receiving a steady stream of complaints from senior paralegals. The AI’s summaries were often "plausible but wrong," citing outdated statutes or misinterpreting the nuances of internal policy documents. User trust was eroding, and the human approval rate for generated summaries was stuck at a costly 71%.
The Lead AI Engineer: He was managing the technical stack and the API budget. He observed that average query latency was climbing, and monthly spend on their primary foundation model had doubled in three months without a corresponding increase in usage. His dashboard showed that the average prompt size was ballooning as developers, following the "more context is better" mantra, were programmatically including entire email threads and multiple tangentially related documents to "help" the model.
The Head of Compliance: She was concerned about the unmanaged risk. An AI confidently citing a superseded version of a compliance policy could create significant liability. The lack of versioning or traceability for the information sources used in each generation made auditing impossible. The risk wasn't just financial; it was reputational.
The core constraint was that the team operated under a fixed budget and could not simply switch to a more powerful, next-generation model with a larger context window. They were locked into their current vendor for two more quarters. The pressure was to improve the reliability and economics of the existing system. Their initial attempts—minor prompt adjustments and adding negative constraints ("do not cite outdated laws")—had yielded no measurable improvement. The problem was systemic, rooted in a fundamental misunderstanding of how LLMs use context. The system treated all information as equally valuable, leading to a noisy, contradictory, and expensive decision environment for the model.
The Approach
The operational question shifted from "Which model performs better?" to a more fundamental inquiry: "How do we govern the information our model sees to make a reliable decision under cost and latency constraints?" This reframing forced a cross-functional approach involving product, engineering, and compliance, with each function owning a piece of the solution. The team architected a three-part context engineering policy, acknowledging the trade-offs at each step.
Policy 1: Implement a Strict Context Budget
The first and most impactful change was to stop treating the context window as an infinite closet and start managing it as a finite, high-cost production asset. The Lead Engineer was tasked with creating and enforcing a "context budget" for their primary 128,000-token model.
The trade-off was clear: a tight budget risked omitting potentially useful but low-signal information, while the status quo of an unbounded budget was demonstrably creating noise, increasing cost, and degrading performance. The team decided that a predictable, constrained context was better than a comprehensive but chaotic one.
The process began with an audit of their highest-volume workflow. They discovered that over 50% of tokens in a typical prompt were from redundant email signatures, conversational filler, and duplicative document summaries. The new policy allocated the 128K window into reserved, non-negotiable blocks:
System Prompt & Core Instructions (8K tokens): This block was reserved for the core system prompt, role definition, output format specifications (e.g., JSON schema), and immutable rules like "cite sources for every claim." This part of the context was locked and version-controlled.
Grounded Knowledge Retrieval (40K tokens): The largest portion was allocated to high-quality, versioned documents retrieved via their RAG system. This included specific statutes, internal policy documents, and directly relevant case law snippets. The RAG system itself was modified to return smaller, more relevant chunks instead of entire documents.
User Query & Session History (32K tokens): This block contained the user's immediate question and a summarized history of the current conversation thread. The key change here was aggressive summarization of older turns to preserve their intent without occupying excessive token space.
Tool Function Signatures & Reserve (16K tokens): The team was beginning to experiment with agentic functions that could call external APIs (e.g., a real-time legal database). They reserved a block of tokens for the function definitions and, critically, for the API responses to be injected back into the context mid-flow. Previously, a full context window would cause these tool outputs to be truncated, leading to cryptic errors.
Generation Headroom (32K tokens): To prevent the model from abruptly stopping its output, they reserved a significant portion of the window for the generated response itself.
Context Budget Allocation by Use Case (Illustrative)
Use Case: Legal Analysis · System & Rules: 10% · Retrieval (RAG): 40% · User Thread: 20% · Tool Reserve: 10% · Generation Headroom: 20% · Rationale: Prioritizes high-quality, versioned legal corpora (RAG).
Use Case: Customer Support · System & Rules: 5% · Retrieval (RAG): 25% · User Thread: 45% · Tool Reserve: 5% · Generation Headroom: 20% · Rationale: Emphasizes conversation history (User Thread) for personalization.
Use Case: Code Generation · System & Rules: 15% · Retrieval (RAG): 20% · User Thread: 35% · Tool Reserve: 15% · Generation Headroom: 15% · Rationale: Balances repo context (RAG) with user intent and tool calls (linters, APIs).
This budgeting was not a one-time fix. The engineering team implemented a context assembly service that enforced these caps, logging overruns and providing analytics on which context source was the most frequent cause of overflow. The policy for eviction was simple: when a block exceeded its budget, the lowest-salience chunks were dropped. Salience was initially defined by semantic relevance to the user's immediate query, but this was later refined to include heuristics that protected foundational documents, such as the oldest message in a thread or a document labeled as "Master Service Agreement."
Policy 2: Mandate a Grounding and Freshness Policy
The Head of Compliance drove the second policy, addressing the risk of the AI using stale information. The core problem was that their RAG system pulled from a document store that was not versioned in a way the AI could understand. An updated policy document could be uploaded, but the AI might still retrieve and cite the old version if its embedding was closer to the query vector.
This led to the concept of RAG freshness debt—the accumulating risk of confident, well-written but factually incorrect answers based on outdated sources. To pay down this debt, two changes were implemented:
Corpus Versioning: Every document ingested into the vector store was stamped with a
version_idand alast_updated_timestampin its metadata. The RAG retrieval logic was updated to not only find the most relevant chunks but to also return their version metadata alongside the content.Freshness SLOs: The compliance and product teams established Service Level Objectives (SLOs) for the freshness of different document types.
RAG Freshness SLOs and Business Impact
Corpus Type: Public Statutes & Regulations · Freshness SLO: < 24 hours · Update Cadence: Daily · Business Risk of Staleness: High: Risk of non-compliant legal advice.
Corpus Type: Internal Corporate Policies · Freshness SLO: < 1 business day · Update Cadence: As needed · Business Risk of Staleness: Medium: Risk of incorrect internal guidance.
Corpus Type: Case Law Database · Freshness SLO: < 7 days · Update Cadence: Weekly · Business Risk of Staleness: Low: Precedent changes slowly.
Corpus Type: User Support Knowledge Base · Freshness SLO: < 1 hour · Update Cadence: Continuous · Business Risk of Staleness: Medium: Risk of poor customer experience.
The application logic now checked the version_id of retrieved documents against a centralized "source of truth" manifest. If a retrieved document was a version behind the official one, the system would either attempt to fetch the new version or, if unavailable, explicitly state in the context that the information might be outdated. Every AI-generated answer was logged with the corpus_version of every source it used, creating an auditable trail that support and compliance could use to trace any reported errors.
Policy 3: Engineer Refusal as a Core Feature
The Product Manager owned the third policy: making refusal a designed feature, not an unexpected failure. Previously, when the RAG system returned no relevant documents, the LLM would often try to "fill the gap" by generating a plausible-sounding but unsubstantiated answer. This was the source of the most dangerous hallucinations.
The new policy required the system to refuse to answer when its evidence fell below a set confidence threshold. The engineering team implemented a check: if the RAG system returned fewer than a minimum number of high-relevance chunks (a tunable parameter), the workflow would bypass the LLM's generation step entirely.
Instead of a vague "I cannot answer that," the system would return a structured JSON object to the user interface:
{
"status": "refused",
"reason": "Insufficient context from verified sources.",
"reason_code": "INSUFFICIENT_EVIDENCE",
"required_information": [
"Please specify the exact policy number (e.g., 'CP-401').",
"Please provide the case citation number."
],
"log_id": "c7a8f9b2-4e1d-4f2a-8b9c-1d2e3f4a5b6c"
}This transformed refusal from a bug into valuable telemetry. The PM could now track refusal_rate as a key metric. A high refusal rate on certain topics pointed directly to gaps in their knowledge base. The required_information field prompted the user to provide the necessary details, teaching them how to interact with the system more effectively. Most importantly, the legal and compliance teams viewed this as a critical risk mitigation feature. A logged, auditable refusal is infinitely preferable to an authoritative-sounding hallucination. Each week, the top two refusal cases were reviewed and added to the evaluation "golden set" to ensure the system continued to refuse correctly under ambiguity.
Figure 1: Context budget by chunk type
Illustrative allocation: system rules, retrieval, user thread, tool reserve. This structured approach prevents low-value context from crowding out critical tool outputs or grounded retrieval results.
The Results
The implementation of these three policies over six weeks yielded immediate and measurable improvements across cost, performance, and quality. The team did not change the foundation model or spend any additional money on infrastructure. The gains came entirely from disciplined context management.
Key Performance Metrics: Before vs. After Context Engineering
Metric: Human Approval Rate · Before Policy (Baseline): 71% · After 6 Weeks: 89% · Change: +18 pts · Business Impact: Reduced manual rework, increased user trust.
Metric: Average Tokens per Query · Before Policy (Baseline): 41,000 · After 6 Weeks: 15,500 · Change: -62% · Business Impact: Drastically lowered API costs.
Metric: API Spend (Core Workflow) · Before Policy (Baseline): $52,000 / month · After 6 Weeks: $19,700 / month · Change: -62% · Business Impact: Freed budget for investment in other features.
Metric: P90 Query Latency · Before Policy (Baseline): 12.4 seconds · After 6 Weeks: 4.1 seconds · Change: -67% · Business Impact: Improved user experience and interactivity.
Metric: Refusal Rate (Ambiguous Queries) · Before Policy (Baseline): 2% (often hallucinated) · After 6 Weeks: 15% (structured refusal) · Change: +13 pts · Business Impact: Increased safety and provided clear data on knowledge gaps.
The most significant outcome was the cultural shift. The team stopped blaming the "model quality" for failures and started taking ownership of the data fed into it. The dashboard for the PM and Lead Engineer now prominently featured metrics like "average context fill percentage" and "refusal rate by topic," turning context engineering into a first-class operational discipline.
What Went Wrong
The transition was not without issues. Our initial eviction policy for the context budget was too simplistic and caused a temporary regression in quality. The first version of the context budget service used a "last-in, first-out" (LIFO) logic combined with a basic semantic similarity score to evict chunks when the user thread buffer was full. For simple Q&A, this worked.
However, for complex, multi-turn legal analysis, this approach proved flawed. The first message in a long conversation often contained the foundational legal question or the core case file that governed the entire interaction. Our LIFO policy was evicting the most important piece of context first. In the first two weeks, our human approval rate on queries involving more than three turns dropped from 71% to 65%. The model was correctly answering the most recent question but missing the overarching context.
We corrected this by implementing a more sophisticated eviction algorithm. The new logic assigned a "protection score" to context chunks based on heuristics:
Chunks from the first user turn received a high score.
Chunks containing specific legal citation patterns (e.g., "U.S.C. §") were protected.
Summaries of previous turns were given lower scores than the original text.
This fixed the regression and highlighted a key lesson: context budgeting isn't just about capping tokens; it requires an intelligent, domain-aware strategy for determining what information is most valuable to preserve.
Key Takeaways
Add a closing paragraph to your context policy documents: refusal rate is a success metric when evidence is missing. Budget token headroom for tools and agentic functions before you chase a vendor's promise of a larger context window. Log the corpus_version on every customer-visible answer so your support teams can instantly reconcile user tickets with policy portal timestamps. Review your context eviction rules whenever human approval rates drop—it often means you are evicting the wrong chunks, not that you need a more expensive model.
Version your corpora like you version your code—RAG freshness is a product SLO that directly impacts user trust and liability.
Refusal is a critical safety feature, not a conversational failure—log it, measure it, and use it to identify gaps in your knowledge base.
What Operators Miss
Teams consistently buy access to larger context windows as a first resort, believing it's a silver bullet for performance issues. Window size does not fix stale retrieval, noisy inputs, or contradictory system prompts. A 1-million-token window filled with irrelevant or outdated information will produce a more expensive and confident hallucination, not a better answer.
Before allocating budget to a model upgrade, operators must instrument their existing workflows. Measure the average context fill percentage, the refusal rate on ambiguous queries, and the human override rate for each critical use case. When the refusal rate climbs immediately after a corpus update, the bug is almost certainly version skew between your index and your source of truth—an issue that no foundation model can solve. The disciplines of budgeting, grounding, and refusal are prerequisites for scaling reliable AI. Pair this memo with RAG Freshness Debt for specific guidance on index SLOs and with AI Engineering for the complete map of production disciplines.
References
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS.
The AI Operator. (2025). AI Engineering — full discipline map.
The AI Operator. (2025). Think Like an AI Operator — prompting for decisions (companion).
OpenAI. (2024). Best practices for prompt and context design (directional). https://platform.openai.com/docs/guides/prompt-engineering
Editorial transparency. Essays at The AI Operator may use AI-assisted research, drafting, and editing tools under staff editorial review. Facts, figures, and recommendations are checked before publication; we correct the record when evidence changes. Questions: hello@theaioperator.net.
Published on [Substack](https://theaioperator2.substack.com/p/context-engineering-memo).