The MRM Memo Your GC Can Audit: Logging Contracts for Prompt + Retrieval Bundles
Board-ready framing for model risk when contracts are the product—five log fields every legal AI deployment must capture before prompt changes ship weekly.
Executive Summary
Board-ready framing for model risk when contracts are the product—five log fields every legal AI deployment must capture before prompt changes ship weekly.
An investment bank reduced AI-driven compliance decision costs by 72% and cut audit preparation time from six weeks to two days by shifting from model-centric MLOps to a version-controlled "MRM Contract" framework. This approach treats every AI component—prompt, data, rules, and model—as a single, auditable unit, forcing P&L ownership and eliminating the risk of unversioned prompts. The framework moved accountability from a central AI team to the business unit P&L, directly linking cost, performance, and risk management.
The Challenge
The operator, a mid-market investment bank, faced mounting pressure in its trade surveillance function. A team of 25 compliance officers was tasked with reviewing alerts generated by a legacy, rules-based system flagging potential market manipulation under regulations like the Market Abuse Regulation (MAR). The system, while deterministic, was brittle. It generated a high volume of false positives—over 98% of all alerts—consuming over 80% of the team's time and driving operational costs up. The daily reality for a compliance officer was a deluge of low-value work, manually cross-referencing trade data with market news and internal communications, leading to high burnout and the constant risk of a true positive being missed due to fatigue.
The Chief Operating Officer (COO), facing both rising headcount costs and pressure from regulators to modernize surveillance capabilities, sponsored a project to deploy a Large Language Model (LLM) to automate the initial analysis of these alerts. The goal was to reduce manual review overhead and allow the expert team to focus on genuinely complex investigations.
The initial proof-of-concept, conducted by the central AI platform team, was successful in a lab environment. A proprietary LLM using a Retrieval-Augmented Generation (RAG) pattern could analyze trade data against internal policies and external news feeds, providing a coherent summary and a recommendation to "clear" or "escalate." The demo was impressive, showing the model correctly interpreting nuanced market events. However, moving to production exposed a critical governance gap between the engineering team, focused on technical performance, and the risk function, focused on legal and regulatory defensibility.
The Head of Compliance and the General Counsel (GC) posed a set of non-negotiable questions that the initial MLOps (Machine Learning Operations) framework, which focused on versioning Python code and model weights, could not answer:
Non-Repudiation: If a regulator questions a trade cleared by the AI six months from now, can we reproduce the exact information and logic—including the specific prompt and retrieved documents—used to make that specific decision?
Change Management: When a prompt is updated by an engineer to improve performance, how is that change reviewed and approved by the compliance team? What happens if the new prompt subtly alters the interpretation of a regulatory rule, effectively creating an unapproved policy change?
Accountability: Who owns the budget and the risk? The cost of the LLM was projected to be significant, yet it was buried in a central "Platform Engineering" budget. This gave the Head of Compliance, the ultimate owner of the surveillance outcome, no control or visibility over the cost-per-decision and no incentive to optimize it.
This is a composite scenario based on our work with operators in three regulated financial services firms. The core conflict was universal: the engineering team was celebrating a 5% increase in the model's F1 score, while the GC and Chief Risk Officer (CRO) were concerned with evidentiary standards and non-repudiation. The CRO, accountable under frameworks like the Federal Reserve's SR 11-7 on Model Risk Management, saw the AI system not as a single model but as a complex, dynamic decisioning process. The existing MLOps system treated the prompt, the RAG knowledge base, and the model as separate, fluid components, creating what the GC termed "unacceptable evidentiary risk.the model was updated" or "the prompt was tweaked" as an explanation for why two similar-looking trades were treated differently. The system, as initially designed, was unauditable.
"An auditor or regulator would not accept."
Figure 3: MRM contract bundle (baseline deploy)
Component: Prompt template · Version artifact: Git tag
prompt/v14· Example hash / ID:sha256:a91f…· Change trigger: Wording or variable changeComponent: Retrieval corpus · Version artifact: S3 object version · Example hash / ID:
corpus/2026-03-18· Change trigger: Policy doc or index refreshComponent: Rules engine · Version artifact: Config bundle · Example hash / ID:
rules/v7· Change trigger: Threshold or allow-list changeComponent: Model configuration · Version artifact: Provider snapshot · Example hash / ID:
gpt-4-turbo-2024-04-09· Change trigger: Model or hyperparameter changeComponent: Bundle output · Version artifact: `contract_version_id` · Example hash / ID: `mrm-trade-surveillance-v14.7` · Change trigger: Any row above changes
Every production inference call had to cite a single contract_version_id so audit replay could reconstruct prompt, corpus, rules, and model state as one unit.
The Approach
The cross-functional team mapped the control path before scaling production traffic.
The core inquiry was not "which model is best?" but "how do we build a system where the cost and risk of every automated decision are non-repudiable?" We had to shift the engineering focus from optimizing model metrics in isolation to building a defensible, end-to-end decisioning apparatus. Standard MLOps practices, focused on versioning model weights and training data, were insufficient because they missed the most dynamic and influential components. The locus of decision logic in modern AI systems has shifted to the prompt template, the retrieval corpus, and the surrounding business rules—all of which were being managed as informal text files, environment variables, or unstructured document stores.
The solution required a shift in perspective: from managing a "model" to versioning an auditable "contract." We defined this Model Risk Management (MRM) Contract as a version-controlled bundle containing every component that influences an AI-driven decision. This wasn't just a logging standard; it was the fundamental unit of deployment and governance.
The Prompt Template: The exact, version-controlled text and variables that structure the query to the LLM. A change from "Does this trade violate Policy 4.1?" to "Critically evaluate this trade against Policy 4.1, noting any ambiguities" is a material change in logic that must be versioned and approved.
The Retrieval Corpus: A version hash (e.g., a Git commit hash or an S3 object version ID) of the documents or database state used for RAG. This ensures that the context provided to the model is as replayable as the prompt itself.
The Rules Engine: A hash of any deterministic business logic applied pre- or post-inference. This includes parsing logic for the model's output, confidence thresholds for escalation, and any hard-coded allow/deny lists that override the model's recommendation.
The Model Configuration: The specific
model_id(e.g.,gpt-4-turbo-2024-04-09),temperature,top_p, and other hyperparameters. These settings control the model's behavior and are integral to the decision's logic.
Each unique combination of these four elements was serialized into a canonical JSON object and then hashed using SHA-256 to create a single, immutable contract_version_id. This ID became the non-negotiable key for logging, audit, and governance. The new mandate from the COO and GC was absolute: no inference call to a production system could be made without a valid, approved contract_version_id. This forced discipline and created a clear "chain of custody" for the logic itself.
Three design choices were critical to implementing this framework.
(1) Immutable Logging as the Source of Truth
The first step was to define a non-repudiable ledger. We established a minimum viable set of seven fields to be logged for every single automated or augmented decision. This logging schema was not a suggestion; it was enforced at the API gateway level, rejecting any request that did not conform. The ephemeral nature of a prompt or a retrieved document was no longer a risk; it was captured by design. Getting agreement on this schema required a cross-functional working group including engineering, compliance, legal, and finance, a two-month process of negotiation to balance granularity with cost.
Figure 4: Log Field vs Data Type vs Purpose & Rationale
The first step was to define a non-repudiable ledger.
Log Field:
decision_id· Data Type: UUID · Purpose & Rationale: A unique identifier for this specific transaction. The primary key for any audit query. Ensures every decision can be isolated and examined. It also serves as the join key to the source system's transaction records.Log Field:
workflow_id· Data Type: String · Purpose & Rationale: The business process name (e.g.,trade_surveillance_l1,client_onboarding_kyc). The primary key for cost allocation and P&L chargeback. This simple tag is the foundation of the economic model.Log Field:
contract_version_id· Data Type: SHA-256 Hash · Purpose & Rationale: The immutable hash of the prompt + retrieval + rules + model bundle. The primary key for risk replay and version control. Links a specific decision to its exact logical context. It answers the question, "What set of rules was in effect at the time?"Log Field:
evidence_bundle_uri· Data Type: URI (S3/GCS) · Purpose & Rationale: A pointer to a stored object containing the exact snippets retrieved and fed into the context window for this specific decision. Essential for reconstruction and debugging. Stored in WORM (Write-Once, Read-Many) storage for a 7-year retention period, satisfying regulatory requirements.Log Field:
model_inference_log· Data Type: JSON Object · Purpose & Rationale: A nested object containingmodel_id,input_token_count,output_token_count, latency, and the raw model response before parsing. Provides a complete record of the model interaction for cost analysis, performance monitoring, and debugging model-specific behavior.Log Field:
human_override· Data Type: Boolean · Purpose & Rationale: A flag indicating if the automated decision was escalated or overturned by a human operator. Critical for measuring the system's true autonomy, calculating straight-through processing (STP) rates, and identifying areas of model weakness for retraining or process improvement.Log Field:
decision_outcome· Data Type: JSON Object · Purpose & Rationale: The final, structured output of the workflow after applying any business rules to the model's response (e.g.,{ "status": "cleared", "confidence": 0.998, "reason_code": "R-42" }). This is the final, auditable result of the process.
This logging contract meant an auditor's query—"Show me exactly why transaction #XYZ was cleared on May 15th"—could be answered with a single database query. The query would retrieve the decision_id, join to the contract_version_id to show the exact logic bundle, and provide a direct link via evidence_bundle_uri to the precise information presented to the model. The audit simulation went from a weeks-long manual scramble involving multiple teams to a minutes-long automated report.
(2) Tiered Inference Routing with Fail-Closed Validators
With clear cost and workflow attribution, we could optimize the economics. The initial plan to use a single, expensive frontier model (like GPT-4) for all transactions was financially untenable and operationally naive. It was like using a sledgehammer to crack a nut for the vast majority of simple alerts. We replaced it with a tiered routing system for the trade surveillance workflow, creating a "logic cascade" that matched cost to complexity.
Tier 1 (Triage): A small, fast, fine-tuned open-source model (a 7-billion parameter model fine-tuned on 100,000 internal, anonymized alert-resolution pairs) processed 100% of incoming alerts. Its sole job was to perform two checks: data completeness and pattern matching against a set of known, low-risk scenarios that had been consistently cleared by human officers in the past. This tier cost less than $0.005 per decision and immediately cleared approximately 85% of the total volume with extremely high confidence.
Tier 2 (Analysis): The remaining 15% of more complex or ambiguous alerts were routed to the powerful proprietary model for sophisticated RAG-based analysis against the full corpus of policies and market data. This tier cost approximately $0.22 per decision, an expense now justified by the complexity of the task.
Fail-Closed Gate (Human Escalation): This was the most critical risk control. If the Tier 2 model's confidence score (extracted from its structured JSON output) was below a 99% threshold, or if the transaction involved entities on a high-risk watch list, the system was architected to fail closed. It did not render a final "clear" or "block" decision. Instead, it packaged the entire context—the raw alert, the retrieved evidence, and the model's draft summary—and routed it to a human compliance officer's queue under a strict 4-hour Service Level Agreement (SLA).
This tiered approach dramatically lowered the blended cost-per-decision. It also provided a deterministic backstop to a probabilistic system, satisfying the GC's requirement for a robust "human-in-the-loop" control for the highest-stakes decisions. The fail-closed principle, borrowed from safety-critical engineering, ensured that uncertainty always resulted in human oversight, not silent failure.
Economic Impact of Tiered RoutingInitial approach (single model):monthly_cost = 1,000,000 alerts × $0.22/decision = $220,000/month
Tiered approach:tier1_cost = (1,000,000 × 85%) × $0.005 = $4,250tier2_cost = (1,000,000 × 15%) × $0.22 = $33,000total_cost = $37,250/month
This represented a cost reduction of over 83% for the inference component of the workflow, a figure that got the immediate attention of the COO and demonstrated the financial power of matching computational expense to problem complexity.
(3) Governance Coupling and Org Design
Technology alone was insufficient; the operating model had to change. A technical framework without corresponding changes to organizational structure and incentives would fail. The MRM Contract was coupled directly to financial accountability and role definition. We moved from a central platform budget—a classic "tragedy of the commons" problem—to a formal chargeback model. The workflow_id in our logs became the billing key. The VP of Operations, who owned the compliance team, now had a formal line item on her P&L for "AI Decisioning - Trade Surveillance," calculated directly and automatically from the immutable logs each month.
This created immediate ownership. Within the first 30 days, her team was asking the platform engineers questions about prompt token counts and retrieval chunking strategies, seeking ways to drive their new cost center down. The moral hazard of the central "free money" pool was eliminated. The conversation shifted from "Can the AI do this?" to "What is the ROI of automating this decision?"
We also formalized the role of the "Contract Owner." For the trade surveillance workflow, the Head of Compliance was designated as the owner. This was not a technical role. Her responsibility was to own the business logic and risk tolerance of the automated system. Any change to the prompt template, the data sources for retrieval, or the business rules in the post-processing engine required her digital sign-off within a Git-based workflow, managed through a simplified user interface.
The deployment process was re-engineered from a purely technical pipeline to a governance-centric one.
Before: An engineer pushes code to a repo -> CI/CD deploys the new model/prompt.
After: An engineer proposes a change to a prompt/rule in a development branch -> A pull request is generated -> The change triggers an automatic generation of a new
contract_version_idand runs a suite of regression tests on a "golden dataset" of past alerts -> The Head of Compliance receives a notification with a clear "diff" showing exactly what logic is changing and the impact on test cases -> She can approve or reject the change with a comment -> On approval, CI/CD deploys the new contract, making it available for production traffic.
The unit of deployment was no longer just code; it was a new contract_version_id that bundled code, configuration, test results, and risk sign-off into a single, traceable artifact. This directly addressed the "hidden technical debt" of entanglement in machine learning systems, where changing one component can have unintended and unmonitored consequences elsewhere (Sculley, et al., 2015) [2].
These choices presented clear trade-offs, which we debated extensively with leadership.
Figure 5: Option vs Upside vs Downside
These choices presented clear trade-offs, which we debated extensively with leadership.
Option: Central Pool · Upside: Fast demos, low initial friction for business units. Encourages experimentation without immediate budget fights. · Downside: Moral hazard, opaque $/decision, high risk of cost overruns, zero risk ownership by the business. Fails basic audit and governance requirements for regulated industries. · Our Decision & Rationale: Rejected. This model is unsustainable at scale. It creates systems that are impossible to govern and financially unaccountable. It is suitable only for early-stage R&D, not production workloads.
Option: Showback · Upside: Provides cost visibility to business units without direct financial impact. Acts as an intermediate step to build cost awareness. · Downside: No direct financial incentive to optimize; costs are treated as "funny money." Can be a useful transitional step but does not solve the accountability problem. · Our Decision & Rationale: Used as a 60-day bridge. This allowed us to build awareness and validate the accuracy of our logging and cost attribution logic before switching to the harder-edged chargeback model. It gave the business unit time to understand the drivers of their new cost base.
Option: Chargeback + Validators · Upside: Creates direct P&L ownership, forces optimization, and makes the system auditable by default. Aligns incentives between engineering and the business. · Downside: Higher initial setup overhead, requires disciplined logging, and demands a cultural shift to business-led governance. Can create friction if not implemented with clear communication. · Our Decision & Rationale: Adopted. This was the only model that satisfied the GC's requirements for accountability and the COO's need for financial control. It aligned cost, risk, and performance at scale, making the system sustainable and defensible.
The Results
The implementation of the MRM Contract framework over six months produced a step-change in performance, cost, and risk posture. The metrics, tracked from the immutable logs, provided the COO and the board's risk committee with a clear, quantitative picture of the outcome. The data was not an estimate; it was a direct product of the system's architecture.
Figure 6: Metric vs Before vs After
The implementation of the MRM Contract framework over six months produced a step-change in performance, cost, and risk posture.
Metric: Average Cost-per-Decision · Before: $0.22 · After: $0.062 · % Change: -72%
Metric: Audit Evidence Retrieval Time · Before: 6 weeks (manual) · After: <2 days (automated) · % Change: -97%
Metric: Alerts Requiring Human Review · Before: 100% · After: 15% · % Change: -85%
Metric: Unreproducible Decision Incidents · Before: 3-4 per quarter · After: 0 · % Change: -100%
\Note: The final blended cost-per-decision of $0.062 is higher than the pure inference cost of $0.037 because it includes amortized costs for logging, WORM storage, compute for the rules engine, and the human review of the 15% of escalated cases. This represents the true, fully-loaded cost of the workflow.
The most significant outcome was not purely financial. The Head of Compliance reported a fundamental shift in her team's focus and morale. Instead of manually clearing thousands of low-risk, repetitive alerts, they were now focused exclusively on the 15% of complex cases escalated by the AI. Their work became more investigative and higher-value, leveraging their deep subject matter expertise. Team attrition dropped, and the group began to function as a true investigations unit rather than a processing center.
Furthermore, the MRM Contract pattern became a reusable asset. The framework was subsequently applied to two other regulated workflows—Know Your Customer (KYC) document analysis and communications surveillance—reducing the development and governance setup time for those projects by over 50%. The initial investment in the framework paid dividends across the organization.
What Went Wrong
Our initial implementation of the RAG corpus versioning was over-engineered and academically pure. We tied the contract_version_id to the SHA-256 hash of the entire document corpus. The bank's internal policy documents and market data feeds, however, were updated frequently with minor grammatical or formatting changes, or the addition of inconsequential news stories. This created "version thrashing," where a trivial change to a single document would generate a new contract_version_id, invalidating the old one and forcing a full regression test and re-approval cycle by the Contract Owner.
For two weeks, the system was generating dozens of new contracts per day. This created significant operational friction and frustrated the Head of Compliance with a constant flood of approval requests for meaningless changes. The process, designed to ensure governance, became the primary bottleneck to operations. The engineering team was spending more time managing contract versions than improving the system's logic.
We corrected this by decoupling the contract from the raw document content. We shifted to versioning the embedding model and the indexing process instead. The logic was that a change in how we interpret documents (the embedding model) is a material change, while a minor change in the documents themselves may not be. The retrieval corpus was now refreshed on a nightly basis, and this batch update was logged with its own version ID, but it no longer generated a new contract_version_id. We accepted a maximum 24-hour data staleness as a reasonable business trade-off for operational stability. High-velocity data, like sanctions watch lists, was handled via a separate, real-time API call that was logged as part of the evidence_bundle but did not version the contract itself. This pragmatic compromise balanced perfect auditability with operational reality.
Canonical scope
Archive note (June 2026): Public canonical for the mrm bundles cluster. Horizontal MRM memo for prompt + retrieval bundles. Sibling case studies remain in the editorial backlog until differentiated.
Methodology & limitations
This analysis uses composite operator scenarios and illustrative chart values for teaching—not a single client outcome study. Adjust for your domain before production decisions.
References
Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://www.nist.gov/itl/ai-risk-management-framework
Monday Morning Checklist
[ ] Assign a named P&L owner for
mrm-memo-prompt-retrieval-bundlesin the Decision Ledger.[ ] Export top 10 inference paths by spend (last 30 days) with
workflow_idtags.[ ] Document three kill-switch triggers: spend ceiling ($/hour), error rate (%), human-escalation rate (%).
[ ] Pilot fail-closed validators on one high-risk path before expanding agent autonomy.
[ ] Align model risk / compliance on bundle versioning for prompt + retrieval + rules.
[ ] Schedule 30-minute review with finance: walk Figure 1 ledger for mrm memo prompt retrieval bundles; agree showback vs. chargeback date.
Editorial transparency. Essays at The AI Operator may use AI-assisted research, drafting, and editing tools under staff editorial review. Facts, figures, and recommendations are checked before publication; we correct the record when evidence changes. Questions: hello@theaioperator.net.
Published on [Substack](https://theaioperator2.substack.com/p/mrm-memo-prompt-retrieval-bundles).