An agent can act on an incorrect answer, so answer accuracy alone is insufficient to assess risk. It sends the email, updates the record, issues the refund. Guardrail design is therefore not a safety add-on — it's the core engineering discipline of agent development.
Patterns to consider for a scoped agent
- Typed tool contracts. Every tool the agent can call has a strict schema, validated before execution. The agent cannot pass a free-text SQL string; it passes structured parameters that the tool validates and executes itself. Validate permissions and parameters before executing a tool call.
- Bounded loops and budgets. Hard caps on steps per task, tokens per task, spend per task, and wall-clock time. An agent that exceeds its budget doesn't try harder — it stops and escalates with its trace attached.
- Permission scopes per action, per tenant. Scope read and write access to the records and actions required for the task. In multi-tenant systems this isolation is non-negotiable: agent A must be architecturally incapable of touching tenant B's data, not merely instructed not to.
- Risk-tiered approval gates. Classify the actions in the agreed workflow: reversible and low-cost runs autonomously; irreversible or expensive queues for one-click human approval with full context. Over time, as measured accuracy on a tier rises, its threshold can be loosened — authority is earned with evidence, like a new hire's.
- Traces as a first-class output. Record relevant decisions and tool outcomes in structured traces, with redaction, access controls and retention appropriate to the data. When an agent misbehaves, the question "why did it do that?" should be investigated using the trace, not reconstructed from guesswork.
Failing safely is a feature
Define and test behaviour for anticipated failures such as tool timeout, ambiguous instruction, low-confidence extraction, exceeded budget. Choose an explicit fallback — usually "summarise what happened and hand to a human" — rather than improvising. Ironically, teams trust agents with more autonomy precisely when the agents demonstrate they know when to stop.
These considerations inform the scope discussed on our agentic AI development page, and scoped automation pipelines with explicit review queues.