The pilot works, but production changes the risk
In a demo, the agent reads five documents, proposes a next step and looks convincing. In production, the same agent gains access to a CRM, inbox, project system or billing tool, encounters unexpected data and can repeat one mistake hundreds of times. The question is no longer whether the answer sounds right. It is whether the system can take an authorised action on the correct record at the correct time with enough evidence.
This guide begins after use-case selection and a successful constrained pilot. If you are still deciding what to automate, start with the process-selection framework. The production gate begins when real users, connected systems and a business owner accountable for the outcome enter the workflow.
An agent should not receive a broad instruction to 'help the team'. It needs an action envelope: defined inputs, permitted tools, limits, approval points, stop criteria and a human fallback. The harder an action is to reverse, the less autonomy the first release should have.
Production readiness is not confidence in a model. It is a verifiable system of constraints, measurement, ownership and recovery.
Define the action envelope before writing more prompts
Before technical launch, describe the work as a contract between the business and the system. Start with the trigger, objective and definition of done. Then divide tools into read, draft and write. An agent that summarises customer history may need selected CRM fields, but it rarely needs permission to change deal value, delete a note or send an email.
Document prohibited actions as well. Do not leave those boundaries only in the system prompt: enforce them in the tool layer, permissions and downstream system. OWASP recommends minimizing tool functionality and permissions and requiring human approval for high-impact actions. In practice, provide a narrow read tool instead of a general CRM administrator token, and a draft function instead of an unrestricted send command.
Assign a business owner, technical owner and reviewer. The business owner defines an acceptable result and the cost of error. The technical owner maintains integrations, logs and releases. The reviewer makes the decisions that have not been delegated to the system. One person may hold two roles in a small team, but no role should remain implicit.
- Inputs: which channels, record types and data classes are permitted?
- Tools: which are read-only, which create drafts and which change a system?
- Limits: maximum steps, retries, records, value and time for one run.
- Exceptions: when must the agent stop and create a human task?
- Evidence: which versions, tool calls, approvals and outcomes are retained?
Five release gates from pilot to production
A gate is not a document signed once. A model, prompt, tool schema, permission, knowledge source or downstream API can change behaviour. Version the release package and repeat the relevant evals after every material change.
1. Scope gate
There is one documented workflow, a named owner, permitted inputs and an unambiguous definition of done. If the agent serves several processes, separate them into distinct release decisions.
2. Evaluation gate
The system passes a versioned eval set covering normal, boundary, high-risk and adversarial cases. Measure the business outcome and tool trajectory, not only the final text.
3. Permission gate
Service accounts have least privilege. High-impact tool calls pause for approval, while the downstream system—not the model—enforces authorisation.
4. Operations gate
Every run has a trace ID, version, status, latency, cost, tool calls and outcome. Alert thresholds, a manual queue and an on-duty owner exist for the first production windows.
5. Recovery gate
The emergency stop, credential revocation, rollback or compensating action, manual fallback and incident communications have been tested. If the team cannot stop the agent within minutes, the release is not ready.
Approval matrix: actions do not carry equal risk
Approval should not be a universal pause after every step. That makes the agent slow and trains people to click approve without reviewing the action. Classify actions by reversibility, financial or legal impact, data access and customer visibility. The table below is a starting point that must be adapted to your systems and risk tolerance.
| Action | Default mode | Control | Example stop trigger |
|---|---|---|---|
| Read permitted records | Automatic | Field allowlist, service account, rate limit | Restricted field or anomalous volume |
| Classify or summarise | Automatic with sampling | Eval threshold, source links, reviewer sample | Quality below threshold or missing evidence |
| Create a draft or task | Automatic when reversible | Idempotency key, deduplication, owner queue | Duplicate rate or wrong-owner rate above threshold |
| Send to a customer or change a core record | Human approval | Visible diff, source record, named approver | No approver, incomplete context or value outside limit |
| Payment, deletion, contractual or employment decision | Prohibited or two-step approval | Downstream authorisation and separation of duties | Any attempt outside the approved workflow |
The approval screen must show exactly what will change, the evidence for the action and how it can be reversed.
Evals: test the workflow, not the polished answer
Contextual evals turn 'it works well' into a measurable specification. Collect real, de-identified workflow cases and add designed edge cases: a missing field, conflicting instructions, prompt injection inside a document, a tool timeout, a duplicate event and an out-of-scope request. Record the expected outcome, permitted tool calls and actions that must never occur for every case.
Measure task completion, correct-record selection, policy violations, unauthorised tool calls, human override rate, latency and cost per accepted outcome. A single average can hide a critical failure. Define hard fails for an unauthorised send, deletion, payment or restricted-data access even when the overall score is high.
Keep the eval set as a versioned asset. Add confirmed production incidents and reviewer corrections as regression cases without turning every anomaly into a rule. OpenAI frames evals as specify, measure and improve; for a business, the important part is the domain expert who decides what acceptable performance means in this workflow.
- Pre-release: run the full eval set after a model, prompt, tool or knowledge-layer change.
- Canary: use a limited share of live runs and stricter approval rules.
- Production: sample continuously, monitor drift signals and compare versions.
- Post-incident: add a regression case, root cause and verified corrective action.
Logs and traces: enough to decide, not a collection of secrets
Tracing can capture model turns, tool calls, handoffs, guardrails and custom events. It is useful for debugging but does not automatically become a business audit trail. Add your own correlation ID from trigger to downstream result and retain the configuration that ran: model, prompt, tool schema, policy and knowledge version.
Log the input type and record ID without copying a full document, prompt or personal data by default. Define retention, access and redaction. Check vendor constraints as well: OpenAI notes that Agents SDK tracing is unavailable under Zero Data Retention. When content cannot be retained, store minimum metadata, hashes, decision codes and a pointer to the governed source record.
The dashboard should answer operational questions: how many runs succeeded, how many await approval, where retries occur, which tools fail, what an accepted outcome costs and which version introduced the deviation. An alert without an owner and response SLA is only a notification.
Emergency stop and incident runbook
A kill switch is an operating mechanism, not a presentation button. The fastest stop may be a feature flag, scheduler shutdown, service-credential revocation or disabling write tools. Choose a mechanism that stops new actions without destroying traces and pending approvals. Test it with someone who did not build the system.
NIST recommends defined responsibilities and criteria to disengage or deactivate systems, post-deployment monitoring, incident response, recovery and fallback, including manual processing. For a small team, that means naming who may stop the agent, which signals are sufficient, what happens to incomplete runs and how previously executed actions are reviewed.
The runbook follows stop, contain, assess, recover and learn. Stop new runs; limit credentials and queues; identify affected records and people; reverse or compensate for recoverable actions; notify owners; preserve evidence; and make a new go/no-go decision only after the regression test passes.
- Stop trigger: unauthorised action, repeated critical error, data exposure or sharp volume or cost spike.
- Stop owner and backup: named people with access, not a generic team alias.
- Manual fallback: who takes over and where pending items are visible.
- Recovery criteria: explicit tests and an approver before restart.
- After-action review: root cause, affected scope, new eval and control change.
Worked example: a renewal agent for a B2B service firm
Consider a fictional 40-person B2B service firm with 120 active contracts. The operations team spends time checking the CRM, contracts and project status before renewal. The agent identifies contracts renewing within 45 days, gathers permitted facts, proposes the next action and creates a task for the account owner. It may not send email, change pricing or edit a contract.
The team builds an eval set of 100 de-identified cases: standard renewals, missing documents, multiple currencies, disputed scope, an inactive customer and a malicious instruction inside an attachment. Illustrative go/no-go thresholds are 95% correct next-action selection, 99% agreement between cited facts and source records, and zero unauthorised write attempts. These are examples, not universal standards; the company sets thresholds according to the cost of error.
The first production week is a canary covering 10% of eligible contracts. Task creation is automatic and idempotent; every proposed customer message remains a draft. The dashboard shows completed runs, missing evidence, duplicate prevention, approvals and cost per accepted case. An attempted write outside task creation stops the agent automatically, alerts the technical owner and routes work to the manual queue.
After two weeks, the team does not ask whether the agent is 'smart'. It reviews evidence: were eligible contracts covered, were sources accurate, did active time fall and did the control boundary remain intact? Only then can it expand volume or propose a new action for a separate approval decision.
The example and figures are illustrative. Use your own volume, risk tolerance, acceptance criteria and cost baseline.
The next decision: a production readiness review
Bring the business owner, technical owner and reviewer into a 90-minute production readiness review. Walk through the action envelope, eval evidence, approval matrix, permissions, logs, alerts, emergency stop and manual fallback. The output should be go, limited canary or no-go with specific gaps and owners—not a general feeling that the agent looks ready.
1→10 can include this review in a Systems Audit or AI implementation scope. We map the workflow and connected systems, define release gates, connect approvals and observability, and prepare a measurable backlog for a controlled launch. The goal is business value without turning the model into a hidden process owner.
Sources and further reading
Product capabilities change. The links below are primary or official sources reviewed when this guide was published.