17 June 20263 min read

Governing AI agents in production: control, accountability, and audit

Making an agent reliable is one problem. Governing a non-human actor with commit access is another: who owns its actions, how far a bad one reaches, and whether you can prove what happened. The governance layer, plainly.

tsukumo

Governing AI agents in production: control, accountability, and audit

Short version: making an agent produce good work is a reliability problem (we covered that in how to make agents reliable). Governing one is a different problem: you've put a fast, non-human actor with commit access into your organization, and you need to control what it can reach, contain what a bad action can damage, keep a human accountable for decisions, and be able to prove later what happened. Governance is the accountability layer on top of reliability, and skipping it is what turns one bad agent run into an incident with no owner.

Reliability versus governance#

They're often confused. Reliability asks does the agent do the work correctly? Governance asks who is accountable for it, how far can it reach, and can we prove what it did? A perfectly reliable agent with unmanaged access is still a governance failure waiting to happen, the same way a competent engineer with unaudited production root access is a risk regardless of skill. You need both, and most teams build the first and forget the second.

Least privilege, by default#

An agent should be able to reach exactly what its task needs and nothing more. No standing broad access, no shared admin token, no production reach for a job that lives in staging. This is ordinary access governance applied to a new kind of account: scope per task, least privilege, and secrets the agent can use without ever seeing. The point isn't distrust of the model; it's that the cost of getting access wrong scales with how fast and autonomously the actor moves, and agents move fast.

A human at the gate for what matters#

Autonomy and approval aren't all-or-nothing. The governing question for any action is: is it reversible, and how big is the blast radius? Cheap and reversible actions can run autonomously. Irreversible or wide-reaching ones (a production deploy, a schema change, anything touching customer data) keep a human approval gate the agent cannot route around. The human isn't typing the work; they own the decision to ship it. That gate is a governance control, not a productivity tax, and it's the practical form of augmenting rather than replacing your team.

Contain the blast radius#

Assume an agent will eventually do something wrong, and design so it can't do much damage when it does. Staging before production. Changes that are easy to revert. Limits on how much an agent can touch in one action. Rate and scope caps so a loop can't run away. Good governance isn't only preventing bad actions; it's making sure the ones that slip through are small and quick to undo.

An audit trail you could defend#

You should be able to answer, weeks later, what did this agent do, and why did this change ship? That requires observability that durably records each action, its cost, and its outcome, not just live dashboards that scroll away. This is what makes agents auditable instead of a black box, and it's what an incident review or a compliance auditor will ask for. Running agents in our own production is why we treat this as non-negotiable, not a nice-to-have.

The standards don't drop because AI did it#

The fastest way to lose the plot is to wave agent output through a lighter process because "the AI wrote it." Agent changes go through the same CI, the same security checks, the same review culture, and the same compliance constraints as anything else, with guardrails the agent can't quietly step around. If anything, a tireless actor that opens many changes raises the bar for your gates, it doesn't lower it. Holding the standard is how you scale agents without scaling risk.

Name an owner#

The thread through all of this is accountability. Every agent has a human owner; every merge that matters has a human approver. When something goes wrong, the answer to "who's responsible" is a person, never "the AI." Governance that diffuses accountability into the tool is the failure mode to design out from the start, and it's part of what an honest readiness check measures.

How tsukumo approaches it#

We run agent fleets in our own production, so we build the governance layer because we need it, not to satisfy a checklist: least privilege, human gates on what's irreversible, contained blast radius, a real audit trail, and standards that hold. When we install the operating model on a client team, these controls come with it, fitted to your stack and your compliance bar, so your team can run agents in production and still answer for everything they do. If putting agents near production worries you, that's the right instinct, and the conversation to have. Talk to us about your team.

Common questions

How do you govern AI agents in production?

Treat the agent as an automated actor with access and govern it accordingly: least-privilege permissions, a human approval gate for irreversible actions, blast-radius limits and easy rollback, a durable audit trail, and the same CI, security, and compliance bar every change meets. Name a human owner for each agent and each merge.

Should an AI agent be allowed to deploy to production on its own?

Not for irreversible or high-blast-radius actions. Keep a human approval gate the agent can't bypass for those, and let it act autonomously only inside a scope where a mistake is cheap and reversible. Expand that scope as trust is earned, in writing.

How do we keep an audit trail of what AI agents did?

With observability that records each action, its cost, and its outcome, durably enough to reconstruct after the fact. The test is whether you could answer 'what did this agent do, and why did this change ship' weeks later, for an incident review or an auditor.

Who is accountable when an AI agent makes a mistake?

A human, always. Governance assigns a named owner to each agent and a human approver to each merge that matters. The agent does the work; a person owns the decision to ship it. Diffusing accountability to 'the AI' is the failure mode to avoid.

Want this running on your team?

Talk to us about your team