Governing autonomous AI agents

What changes when the AI can act

With a model, the control point is the output and the person who uses it. With an agent, the control point is the action: a sequence of tool calls that can write to systems, send communications, move records and trigger downstream processes.

That shifts the governance question from "is this output accurate enough" to "what is this identity permitted to do, under what conditions, with what oversight, and how would we know if it went outside those limits".

Give every agent a distinct identity

Agents must not run under a shared service account or a human's credentials. Each agent needs its own identity, its own least-privilege permission set, and its own accountable business and technical owners.

Without distinct identity there is no attribution, and without attribution there is no meaningful audit record of what the agent did versus what a person did.

Declare the tool inventory

• Every tool, API and data source the agent may call
• Read versus write classification for each one
• Data categories reachable through each tool
• Rate, volume and value limits per tool
• Environments in which each tool is permitted
• The owner accountable for each permission grant

Set autonomy tiers explicitly

Autonomy should be a declared tier, not an emergent property. A workable ladder is: suggest only; act in a sandbox; act in production with prior approval per action; act in production within pre-approved limits with post-hoc review; and act without approval, which should be rare and tightly bounded.

The tier drives everything else — risk band, control set, review frequency and the level of monitoring required — so it must be recorded, approved and enforced technically rather than described in a document.

Put approval gates on consequence, not on volume

Approval gates should be triggered by the nature of the action rather than by how often it happens. Irreversibility, external commitment, personal-data disclosure, financial movement and changes to access rights are all gate conditions.

Gates only work if they fail closed. If the agent can proceed when the approval service is unavailable, the gate is advisory.

Monitor behaviour, not just uptime

• Tool-call volume and distribution against expected baselines
• Escalation and human-override rates over time
• Blocked-action attempts and boundary violations
• Loop, retry and cost anomalies
• Prompt-injection and untrusted-content indicators
• Drift in the tasks the agent is actually being asked to do

Evidence every action

The record of an agent is its action log. Each invocation should be captured append-only with the agent identity, the initiating context, the tool used, the outcome, the approval that applied and the policy version in force at that moment.

That record is what allows an organisation to answer, months later, precisely what the agent did and on what authority.

Frequently asked questions

How is agent governance different from model governance?

A model produces an output. An agent takes actions across systems, chains steps, and can create real-world consequences without a person in the loop. Governance therefore has to control identity, permissions, autonomy limits, approval gates and runtime behaviour, not just the quality of a single output.

What should an AI agent never be permitted to do unattended?

Irreversible or externally binding actions: moving funds, contracting, disclosing personal or confidential data, changing production configuration, altering access rights, or communicating decisions to affected individuals. These belong behind an explicit human approval gate.

How do you evidence what an agent did?

Each agent needs a distinct identity, a declared tool inventory, and an append-only record of every invocation: which tool, on whose behalf, with what inputs, what result, which approval applied and which policy version was in force at the time.

What triggers a review of an agent already in production?

A new tool or data source, a broadened scope of action, a change in autonomy tier, a model or vendor change, an increase in error or escalation rate, or any incident where the agent acted outside its intended boundary.