AI Agent Governance: How to Control AI Agents in Real Operations

Short answer. AI agent governance is the set of controls that decide what an AI agent can see, what it can do, who signs off and how each action is recorded. In operations, the control that matters most is structural: the agent works inside a case where written rules make the decision and people take the exceptions.

Updated 12 November 2026. Descriptions of other vendors come from their own sites when we checked. Floowed is our product, so read those sections knowing who wrote them.

What AI agent governance means in practice

Most definitions say roughly the same thing: the policies, access controls and monitoring that keep agents safe, within the rules and doing what the business intended. The useful test is what it means on an ordinary afternoon, when an agent has just read a supplier's bank letter and is about to update a vendor record.

At that moment, governance comes down to four questions:

1. Access. What was the agent allowed to read, and which systems could it write to?
2. Authority. Who or what made the decision: the model, a written rule, or a person?
3. Escalation. When the case didn't fit, who got it, with what reason, and did they answer?
4. Evidence. Can you show all of that a year from now, for this exact case?

If you can answer those four for every case an agent touches, you have agent governance. If the answers live in prompt text, chat logs and someone's memory, you have a policy document.

Why agents need different governance than models

Classic AI governance grew up around models such as a risk score or a classifier. You governed the model's training, testing and approval for use, and a person acted on its output. Agents act themselves, often across several systems. Real operations make that harder in three ways.

• The work is long-running. A supplier onboarding or a client handover doesn't finish in one run. It waits on a bank letter, a signed contract, an approver on leave. Governance has to hold across every handoff over those days or weeks, not just inside one prompt.
• Models are probabilistic. The same input can produce a slightly different output on a different day. That's fine for drafting a reminder. It isn't fine for deciding whether a supplier gets paid. Businesses need deterministic controls around probabilistic outputs.
• Actions cross systems. An agent that can write to your ERP, your CRM and your e-signature tool has more reach than most of your staff. Access has to be scoped per step, not granted once for the whole agent.

Scott Moore, Floowed's co-founder and CEO, has made this argument directly: the business value of AI comes from moving beyond individual agents and tasks to systems that manage complete, long-running operations. Those systems need persistent case state, governed access to data and an auditable record of decisions and actions. Seen that way, governance is the condition for letting the agent work at all. If you're still choosing the agents themselves, our comparison of AI agents for operations covers the options.

The four layers of AI governance, and what each one covers

Search for "AI governance platform" or "AI governance software" and you'll find very different products under one label. It helps to split them into layers. Most companies running agents in production end up needing more than one.

If you're searching for an AI governance platform because a regulator or your board wants an inventory, risk tiers and policy evidence for every AI system, you want layer 1. Gartner published its first Magic Quadrant for AI Governance Platforms in June 2026, and vendors including OneTrust and IBM reference it on their pages. Floowed isn't one of those platforms and doesn't try to be.

If your problem is that agents already do work in operations and nobody can say who approved what, you need layer 4. That's where the rest of this guide goes.

Seven controls that make AI agent governance real

1. Let rules decide, not the model

The model should read, extract, classify and draft. The decision should come from a written rule, such as "approve the supplier when the registered name, the bank account name and the tax ID all match". In Floowed, rules run in a deterministic policy engine, so the same inputs on the same rule version give the same outcome every time. The model's output is an input to the rule, never a stand-in for it.

2. Scope what each step can read and write

An agent that onboards suppliers doesn't need payroll data. A step that drafts a reminder doesn't need write access to the ERP. Scope access per operation and per step, and give agents their own scoped access instead of running them on an admin's credentials. In Floowed, permissions are set per operation: who builds it, who approves, who publishes changes. Access to customer data is protected by role-based access control, least privilege by default and scoped access tokens.

3. Design the exception path before you need it

Every rule meets cases it doesn't cover. Decide up front where they go. A bank detail mismatch goes to finance. A price variance goes to the budget owner. A document that can't be read with confidence goes to someone who can look at the page. The person should get the case, the evidence and the specific rule that didn't pass, in the place they already work: Slack, Microsoft Teams or email.

4. Put approvals where the risk is

Not every case needs a person, and some need one every time. Floowed lets you set this per operation: a person decides every case, only the ones outside the routine, or none. Most teams start with a person confirming everything and move the line as the record shows it's safe. We cover how to draw that line in human in the loop automation.

Irreversible actions deserve a harder line. Releasing a payment, changing bank details, sending a contract for signature: keep a person or a strict rule in front of those, even after everything else runs on its own.

5. Version every rule, and test before it goes live

Governance fails quietly when someone edits a rule on Thursday and nobody can tell which cases ran on the old version. Every change should be a new version with tests, and you should be able to see what it would have done before it's live. In Floowed, every change to an operation is a new version. You run your tests and replay the change on past cases before you publish it, so you learn, for example, that it would have changed three of your last ninety cases before it goes live, not after.

6. Keep one record per case

For every case, keep what was read and from where, who was asked for what and when they answered, which rule and which version decided it, whether a person confirmed or overrode it and who that was, and what was written to which system. Floowed logs decisions, configuration changes and overrides, attributable to the acting user, with version history. That's what turns "the agent did it" into an answer an auditor accepts. Our guide to the workflow audit trail covers what that record should hold, field by field.

7. Choose models per step, on purpose

Not every step needs the largest model, and not every piece of data should go to every model. Choose models by capability, cost, latency and data requirements, and record which one ran. Google, through Google Cloud Platform, is Floowed's main model provider today, and model-based processing runs under the same access, logging and encryption controls as the rest of the platform. It routes between open-weight and frontier models on cost, latency and data residency.

Worked example: governing an agent in supplier onboarding

Here's how those controls play out on one case. It's an illustration of how the operation runs, not a customer story.

A wholesale distributor adds a new packaging supplier. The buyer starts the case from Slack.

1. Gather. Floowed emails the supplier for the registration certificate, the bank letter and the insurance certificate, and prepares the supplier record for Xero. Nothing goes into Xero until the rules pass or a person approves.
2. Interpret. A model reads the bank letter. The account name is "Harbour Pack Trading". The registration certificate says "Harbour Packaging Ltd".
3. Decide. The rule says the bank account name must match the registered name. It doesn't, so the rule fails. Nobody asks the model whether the difference is probably fine.
4. Chase. Floowed asks the supplier for a bank letter in the registered name and tells the buyer in Slack what the case is waiting on. Two days pass without a reply, so it chases again on the schedule the operation sets.
5. Escalate. The supplier replies that Harbour Pack Trading is its trading name. That's outside the rule, so the case goes to finance with both documents and the failed rule attached. Finance asks for proof of the trading name, which arrives the next morning.
6. Decide again. Finance accepts the exception. The override is logged under their name.
7. Act. The supplier is created in Xero and the buyer is told the first PO can go out.

Four days, three people, one record. If an auditor asks next year why a supplier was set up with a bank account whose name didn't match its registration, the answer is one case, not a search through three inboxes. More on this operation in our guide to supplier onboarding software.

Worked example: an invoice exception

An invoice arrives as a PDF. A model extracts the supplier, the PO number and twelve line items. One unit price is 9% above the PO.

• The rule. Approve automatically when every line is within 2% of the PO and a goods receipt exists. Otherwise, route to the budget owner.
• The model's job. It drafts a short note asking the supplier to confirm the price. The rule, not the model, decides the invoice can't be approved yet.
• The person's job. The budget owner approves the higher price in Microsoft Teams, with "surcharge for expedited delivery, agreed with operations" as the reason.
• The record. Each extracted value with the page and line it came from, the rule version that routed it, the supplier's reply, the approval and the message that came with it, and the vendor bill written to NetSuite.

If someone later moves the tolerance from 2% to 5%, that's a new rule version. Replaying it on last quarter's invoices shows which ones would have gone straight through, before anyone publishes it. If you're choosing tools for this job, our guide to invoice approval software compares the options.

A practical AI agent governance framework

The NIST AI Risk Management Framework organises AI risk work into four functions: Govern, Map, Measure and Manage. It's a good structure for a company-wide program. For the operations agents actually run, here's a shorter path you can start this month.

1. List operations, not agents. Agents come and go. Operations stay: supplier onboarding, invoice exceptions, client handover, contract renewals. Govern the operation, and every agent that works inside it inherits the controls.
2. Write down the rules you already apply. They usually sit in a procedure document or in one senior person's head. If a rule can't be written down, it's a judgment call, and it belongs to a person.
3. Sort actions by how reversible they are. Reading and drafting are low risk. Updating a CRM field is medium. Creating a vendor, changing bank details or releasing a payment is high. Put a person or a strict rule in front of the high ones.
4. Name an owner for each type of exception. A named role that receives the case with the reason, not a shared queue that everybody can ignore.
5. Run in parallel first. Let the system recommend while your team decides, and compare the two case by case before any case closes on its own.
6. Read the record every month. Look for rules people keep overriding. When the same rule gets overridden again and again, the rule is wrong, not the people. Fix it, replay it on past cases and publish a new version.

Roughly, steps 1 and 3 are NIST's Map, 2 and 4 are Govern, 5 is Measure and 6 is Manage, each producing something an operations team can see and change.

How to evaluate AI agent governance software

Ask these in any demo, and ask to see a case that failed a rule, not a clean one.

• Can a model output trigger a write to one of your systems without a rule allowing it?
• For a case closed last month, can you show which rule version decided it?
• Can you test a rule change on past cases before it goes live?
• Who can publish a change, and is that separate from who can build one?
• When a person overrides a decision, are their name and reason recorded?
• Does each agent get its own scoped access, or does it run on a user's credentials?
• Where is data hosted, which model providers process it, and is that in a published sub-processor list?
• Which security certifications does the vendor hold today, and which are in progress? Ask for the report, not the badge.

Where Floowed fits

Floowed runs your operational cases from intake to outcome. Your rules decide, your people handle the exceptions, and every step is on record.

That makes it governance inside the operation, not a governance program around it. Floowed doesn't inventory every AI system in your company, score model risk or produce EU AI Act documentation. If you need that, use a layer 1 platform from the table above, and use Floowed for the operations where agents do the work.

What it does:

• Every case runs five stages: Gather, Interpret, Chase, Decide and Act. A case can wait minutes or weeks between them and keeps its state the whole time. How that works over long cases is in our guide to AI agent orchestration.
• Rules run in a deterministic policy engine, written in plain words or code, with every version kept.
• Exceptions go to the right person, in Slack, Microsoft Teams or email.
• Decisions, configuration changes and overrides are logged, attributable to the acting user, with version history.
• The platform is hosted on Google Cloud Platform, and private deployment is available. Access is role-based with least privilege by default, and data is encrypted in transit and at rest.
• Other agents work with it through the API and MCP in both directions. Your agents can open cases and check their status, and Floowed can hand a step to an agent and wait for the result.

We take security and data privacy seriously. We're working toward SOC 2 and don't hold a SOC 2 report yet. The details are on our security page.

Pricing is credits, not seats. You start with $80 of free credits, no card and no sales call, and Team plans start at $100 a month. See pricing.

Frequently asked questions

What is AI agent governance?

AI agent governance is the set of controls that decide what an AI agent can access, what it can do, who approves its work and how each action is recorded. In operations, it means written rules make the decisions, people handle the exceptions, and every case keeps a record of what happened and why.

How is AI agent governance different from AI governance?

AI governance usually covers the whole AI estate: an inventory of models and systems, risk assessments, policies and regulatory evidence. AI agent governance focuses on agents that take actions across systems. It adds scoped access per step, decision rules, approval points, exception routing and a record of every action.

What is an AI governance platform?

Software for running an AI governance program: discovering AI systems, assessing risk against frameworks such as the EU AI Act, the NIST AI RMF and ISO/IEC 42001, managing policies and producing evidence. OneTrust and IBM watsonx.governance are examples. They govern whether AI may be used. Operational tools govern how each case is decided.

Can AI agents be made deterministic?

The model can't. Models are probabilistic. What you can make deterministic is the decision: the model reads and drafts, and a versioned rule decides. With the same inputs and the same rule version, the outcome is the same every time, and you can replay a rule change on past cases before it goes live.

Do AI agents need a human in the loop?

For some actions, always. For others, only when a case falls outside your rules. Set it per operation, start with a person confirming every case, and widen what runs on its own as the record shows it's safe. Keep a person or a strict rule in front of irreversible actions such as payments or bank detail changes.

What should an AI agent audit trail record?

What the agent read and from where, which model ran, who was asked for what and when, which rule and version decided, any human approval or override with the reason, and what was written to which system. Every entry should be attributable to a person, an agent or the system, with a timestamp.

Is Floowed an AI governance platform?

No. Floowed is an operations platform that governs how each case runs, from intake to outcome. It doesn't replace a GRC or AI governance suite. Many teams will use both, the suite for the AI program and Floowed for the operations agents work in.

Where to start

Pick one operation where an agent already helps, or where you'd like one to. Write down the rules, name the owner for each exception, and run it with a person confirming every case for the first couple of weeks. Start free with $80 of credits, or talk to us and we'll build the first operation with you.