---
Title: "AI governance: how to control what your AI agents can do"
Url: "https://devrev.ai/blog/ai-governance"
Published: "2026-07-27"
Last Updated: "2026-07-27"
Author: "Akhil Kintali"
Category: "AI Quality & Trust"
Excerpt: "82% of enterprises have discovered AI agents they didn't know existed. Here's the 5-pillar framework for controlling what agents can do before they act — not after."
Reading Time: 12
---

# AI governance: how to control what your AI agents can do

An AI agent sent a customer the wrong refund. You know because the audit dashboard flagged it, an hour later. The problem is that knowing an agent sent a wrong email does not unsend it. Most of what gets sold as AI governance today watches agents after they act. That is monitoring. It is not control.

AI governance is the discipline of deciding what your [AI agents](https://devrev.ai/blog/what-are-ai-agents) can do before they act, then enforcing those limits at the moment of action. For a CISO or a VP of engineering facing a board that wants agents in production this quarter, that distinction is the whole game. This guide lays out an operational definition, a five-part framework you can enforce, a procurement checklist, and a maturity model to see where you stand.

## **TLDR**

- AI governance is control over what agents can do before they act, not a compliance report you read after an incident.
- Post-hoc monitoring records failures. Governance-by-design prevents them by enforcing limits at runtime.
- The five operational pillars are permission boundaries, authorization workflows, audit trails, version control, and scope limits.
- Prompt-level guardrails are jailbreakable. Real governance enforces permissions at the data layer, not in agent instructions.
- Only 13% of organizations believe they have the right AI agent governance in place, according to [Gartner](https://tech.co/news/gartner-steps-governing-ai-agents), even as agent counts climb into the thousands.

## **What is AI governance?**

AI governance is the set of policies, controls, and enforcement mechanisms that determine what an AI system can do, who is accountable for its actions, and how every action is recorded. [Gartner](https://atlan.com/know/gartner/ai-governance/) defines it as the process of assigning and assuring organizational accountability, decision rights, risks, policies, and investment decisions for applying AI.

That definition matters because it is operational, not aspirational. A governance program is not a policy PDF or an ethics statement. It is a working control that answers three questions about every agent you run: what can it touch, what can it do, and can you prove it stayed inside those limits. If your program cannot answer those questions with evidence, you have paperwork, not governance.

The stakes scale with the agent count. Gartner predicts the average global Fortune 500 company will run more than [150,000 AI agents by 2028](https://tech.co/news/gartner-steps-governing-ai-agents), up from fewer than 15 in 2025. Each ungoverned agent is an unmapped path into your data and your systems.

## **Why post-hoc monitoring isn't governance**

Picture a support agent with permission to issue refunds. A customer asks for a refund on a $4,000 order. The agent misreads the request and approves it in full when only a $40 credit was warranted. A monitoring tool captures the event, timestamps it, and surfaces it on a dashboard. The money is already gone.

This is the structural flaw in treating governance as observation. Monitoring is retrospective by design. It tells you what happened after it happened, which was adequate for deterministic software and is insufficient the moment reasoning systems are allowed to act. Agents can be redirected by a cleverly worded input, drift as their data changes, and take irreversible actions in seconds. Post-hoc systems observe those failures. They do not prevent them.

The visibility gap is not hypothetical. The [Cloud Security Alliance](https://cloudsecurityalliance.org/press-releases/2026/04/21/new-cloud-security-alliance-survey-reveals-82-of-enterprises-have-unknown-ai-agents-in-their-environments) surveyed 418 IT and security professionals in January 2026. It found that 82% of enterprises had discovered AI agents running in their infrastructure that security or IT did not know about. And 65% had an AI agent incident in the past year. Every organization that had an incident reported real business impact, most often data exposure. You cannot monitor your way out of a problem you cannot see. You certainly cannot govern agents you did not know existed. That is why [monitoring agents in production](https://devrev.ai/blog/ai-agent-observability) sit alongside governance, not in place of it.

**Strategic takeaway:** if your only control is a dashboard, your governance program starts working the instant after it is too late.

## **Post-hoc monitoring vs governance-by-design**

The difference is where the control lives. Monitoring lives outside the agent and reacts to what it did. Governance-by-design lives in the infrastructure the agent runs on and constrains what it can do in the first place.

![Post-hoc monitoring vs governance-by-design](https://cdn.sanity.io/images/umrbtih2/production/9c70ea62a721b2ea19a81405a0fcd6d7f960caa7-1200x660.jpg)

| Dimension | Post-hoc monitoring | Governance-by-design |
| --- | --- | --- |
| When it acts | After the action completes | Before the action executes |
| What it does | Detects and logs failures | Prevents unauthorized actions |
| Enforcement point | Dashboards and alerts | Data layer and action gates |
| Response to a bad action | Investigate and remediate | Block or require approval |
| Irreversible actions | Recorded, not stopped | Gated behind explicit approval |
| Regulatory posture | Evidence of what went wrong | Evidence of control |

**In short:** monitoring tells you an agent broke a rule; governance-by-design stops it from breaking the rule at all. You need both, but only one of them prevents the incident.

## **The 5 pillars of operational AI governance**

Operational AI governance rests on five controls that are enforced by infrastructure, not by agent instructions. Each answers a specific question about agent behavior, and each has to be structural to hold up under an audit. Here is how to implement them.

![The 5 pillars of operational AI governance](https://cdn.sanity.io/images/umrbtih2/production/1b3e524f60c74d47bcf3b1d938eca78fb97fc51c-1200x560.jpg)

### **1. Permission boundaries**

A permission boundary defines exactly what data an agent can read and what systems it can touch. The rule is least privilege: an agent gets the minimum access its task requires and nothing more. An agent that handles order intake should not be able to read executive contracts or change pricing.

The critical detail is where the boundary is enforced. Prompt-level instructions like "do not access financial data" are suggestions a determined user or a misdirected agent can bypass. Real permission boundaries are enforced at the data layer, inherited from your existing identity provider and systems of record, so the agent never sees what the acting user cannot see.

### **2. Authorization workflows**

Authorization workflows decide when an agent must pause and ask a human before it proceeds. This is [human-in-the-loop](https://devrev.ai/blog/human-in-the-loop-ai) control, and it belongs at points of irreversibility: payments, external customer communications, permanent record changes, and any action with regulatory weight.

The skill is placing checkpoints only where mistakes cannot be cheaply undone. Requiring approval for everything recreates the manual process you were trying to automate. As [IBM](https://www.ibm.com/think/insights/liability-laundering-problem-human-in-the-loop-not-governance-strategy) argues, a human in the loop is not a control on its own; a human who can, will, and demonstrably does intervene is. The workflow has to give that person real evidence and real authority, not a rubber stamp.

### **3. Audit trails**

An audit trail is an immutable, append-only record of every action an agent takes: the identity acting, the input, the decision, the tool called, and the outcome. Agents must never be able to modify their own audit records.

This is not only a compliance requirement. It is the foundation for debugging and for trust. When an agent denies a refund or flags a renewal as at risk, the audit trail is what lets you reconstruct the reasoning and defend the outcome to a regulator. Regulated industries increasingly require this level of traceability for automated decisions.

### **4. Version control**

Agents change. Their instructions, their skills, and their permissions all evolve as teams iterate. Version control tracks every change and lets you roll back to a known-good state when a new configuration misbehaves.

Without it, an agent update is a one-way door. With it, teams are free to experiment because they know they can always revert. Automatic versioning turns governance from a brake into an enabler.

### **5. Scope limits**

Scope limits separate what an agent can decide from what it can reach. [Gartner](https://www.beri.net/article/2026-05-31-gartner-40-percent-decommission-tiered-agent-governance) draws the distinction most enterprises conflate: autonomy is what the agent can do, and scope is what data, systems, and permissions it can touch. A read-only agent with access to the entire customer database is a very different risk from a transactional agent with narrow scope, and uniform governance flattens that difference.

Gartner predicts that by 2027, [40% of enterprises will decommission autonomous agents](https://www.beri.net/article/2026-05-31-gartner-40-percent-decommission-tiered-agent-governance) because governance gaps surface only after a production incident. Tiering agents by scope and risk is how you avoid becoming that statistic.

## **Procurement-ready governance checklist**

Buying committees need a shared standard, not a feature list. Use this checklist to evaluate any [AI agent platform](https://devrev.ai/blog/best-ai-agent-builder) against operational governance, not marketing claims.

1. Are permissions enforced at the data layer, inherited from your identity provider, rather than through prompt instructions?
2. Does every agent have a distinct identity, so its actions are attributable to it and to the user it acted for?
3. Can the platform require human approval before irreversible or high-risk actions execute?
4. Is there an immutable, append-only audit trail of every data access and action?
5. Can you roll back an agent to a previous version after a bad change?
6. Can you scope each agent to the minimum data and tools its task requires?
7. Can you terminate a misbehaving agent immediately, and is there a documented decommissioning process?
8. Does the platform ship with SOC 2, role-based access control, and regulatory alignment built in, not bolted on?

If a vendor cannot answer these clearly, they are selling monitoring dressed as governance. The gap is real. Research cited by [Insentra](https://www.insentragroup.com/us/insights/not-geek-speak/generative-ai/agentic-ai-takes-the-wheel-a-deep-dive-into-2026/) found that 63% of organizations cannot enforce purpose limitations on AI agents, and 60% cannot terminate a misbehaving agent quickly.

## **Governance as a deployment mechanism**

Most teams treat governance as a gate they clear before shipping. That framing is backward, and it is expensive. When governance is a separate review step, every new agent deployment triggers a manual risk assessment, ad-hoc evidence collection, and framework-by-framework compliance work that scales linearly with your agent count. The result is what practitioners call governance debt: shortcuts that come due the moment a regulator or an incident forces the question.

The reason most platforms create this debt is architectural. They add AI on top of existing systems and then try to retrofit controls. That produces a bolt-on tax: permissions re-implemented inside the AI layer, and guardrails written as prompts. Prompt-level guardrails can be jailbroken. Data-level permissions cannot. If your controls live in agent instructions, your governance is only as strong as the next clever input.

DevRev built Computer to make governance the deployment mechanism itself. Computer Memory, its permission-aware knowledge graph, inherits access controls from your systems of record and enforces them at the node level.

If Salesforce says a rep cannot see executive data, Computer's agents never see it either. They cannot leak what they cannot access, no matter what prompt they receive. In [Agent Studio](https://devrev.ai/blog/agent-studio), the no-code environment for building agents, Safe Actions makes every action permission-aware, auditable, and reversible before it fires. Confirmation gates sit in front of irreversible actions like refunds and record writes. Automatic versioning tracks every change, and audit logs ship with the platform rather than as a later integration. Agents run inside the same system that holds your data, structured through [how AI knowledge management organizes governed context](https://devrev.ai/blog/ai-knowledge-management). So the five pillars are not features you assemble. They are how an agent is deployed.

This is not theoretical. BILL is a financial operations platform serving more than 500,000 small businesses. It evaluated over 15 AI providers before deploying Computer, and now resolves [70% of support issues autonomously while saving more than $5M](https://devrev.ai/customers/bill), without loosening its controls. Governance and results are not a trade-off when governance is the architecture.

**Strategic takeaway:** if governance is a step you add, it becomes debt; if it is how agents are built, it becomes leverage.

## **AI governance maturity model**

Most maturity models converge on the same four stages. Use this to place your organization honestly, then to see what the next level requires. Roughly 14% of organizations still sit at the foundational level, according to a [Witness AI analysis](https://witness.ai/blog/ai-governance-maturity-model/) of enterprise governance capability.

| Level | Stage | What it looks like | The risk |
| --- | --- | --- | --- |
| L1 | Ad hoc | No formal policy, shadow AI is rampant, no agent inventory | If a regulator asks where your AI is, the answer is a shrug |
| L2 | Defined | Policies exist on paper, enforcement is inconsistent, partial inventory | Controls depend on individual diligence, not infrastructure |
| L3 | Managed | Standardized processes, named owners, risk tiers, audit trails in use | Governance works, but is still partly manual |
| L4 | Optimized | Continuous, runtime-enforced, automated, board-visible | Governance is embedded in how agents run, not checked after |

The jump that matters is L2 to L4. At L2, you have a policy document and hope. At L4, the policy is enforced at runtime, so an agent that would violate a rule is blocked, not merely logged. Governance-by-design is what moves you there, because it converts written policy into structural control.

## **Governance vs security vs observability**

These three disciplines get used interchangeably, and conflating them leaves gaps. They answer different questions.

- **AI governance** is control: what an agent is allowed to do, and enforcing those limits before it acts.
- **AI security** is protection: defending agents and your systems from external threats and misuse. It is the [external-threat side of the picture](https://devrev.ai/blog/ai-agent-security), covering the data, action, and output surfaces an attacker might target.
- **AI observability** is visibility: seeing what agents actually did, so you can debug, improve, and detect drift. This is where [monitoring agents in production](https://devrev.ai/blog/ai-agent-observability) lives.

The clean way to hold them apart: governance decides the rules, security defends the perimeter, and observability watches the behavior. Shadow AI, where teams adopt agents outside IT's visibility, is a symptom of weak governance, and platforms that make AI use auditable turn [shadow activity into governed team work](https://devrev.ai/blog/multiplayer-ai). 

You need all three, but only governance stops a permitted-but-wrong action before it lands. For a deeper look at how these criteria separate real agents from chatbots, DevRev's guide to [evaluating AI agents](https://devrev.ai/blog/evaluating-ai-agent) puts governance among the five tests that matter.

## **Govern agents before they act, not after**

Every governance program eventually gets tested by a single moment: an agent is about to do something it should not. The only question that matters is whether your controls stop it or merely record it. Monitoring writes the incident report. Governance-by-design prevents the incident. The organizations that scale agents safely are not the ones with the best dashboards. They are the ones who decided that governance is not a layer they add. It is how the agent was built.

> [!INFO]
> If you are evaluating how to put real controls around your agents, see how DevRev enforces permissions, approvals, and audit trails at the data layer in [Computer’s Agent Studio](https://devrev.ai/agent-studio).





## FAQ

### What is an AI governance framework?

An AI governance framework is the documented set of policies, controls, review processes, and accountability structures an organization uses to manage how its AI systems are built, deployed, and monitored. The most widely referenced frameworks are the NIST AI Risk Management Framework, ISO/IEC 42001, and the EU AI Act, and most enterprises use a combination rather than one alone. A framework gives you the structure; enforcement at runtime is what makes it real.


### Does the EU AI Act require AI governance?

The EU AI Act sets binding obligations for high-risk AI systems. Its enforcement powers activate on August 2, 2026, with penalties up to €35M or 7% of global revenue. Whether an agent is high-risk depends on its application, such as hiring, credit, or critical infrastructure, not its underlying model. Governance controls like audit trails, human oversight, and traceability are how you demonstrate compliance if an agent falls in scope.


### How do you control what an AI agent is allowed to do?

You control an agent through three things: permission boundaries enforced at the data layer, authorization workflows for high-risk actions, and scope limits that separate what the agent can decide from what it can reach. The key is enforcing these in infrastructure, not in prompts. Prompt-level rules can be bypassed; data-level permissions cannot.


### What's the difference between AI governance and AI security?

AI governance controls what an agent is allowed to do and enforces those limits before it acts. AI security protects agents and systems from external threats and misuse. Governance is about staying inside your own rules; security is about keeping attackers out. A platform needs both, because a perfectly secured agent can still take a permitted action that violates policy.


### How do you govern shadow AI agents?

You govern shadow AI by first discovering it, since you cannot control agents you do not know exist. Build a centralized inventory of every agent across sanctioned and unsanctioned tools, assign each an owner, and route AI adoption through a platform that makes usage auditable and permission-aware. The goal is to convert invisible individual use into governed, visible team activity. Blocking tends to push usage further into the shadows.