How to Govern Thousands of AI Agents

How to Govern Thousands of AI Agents

Govern thousands of AI agents by treating them as an operated workforce of nonhuman identities, not as a large prompt library.

Every production agent needs a registered owner, bounded purpose, unique identity, approved context, controlled tools, action limits, evaluation record, observable runtime, recovery path, and retirement process. Those controls must work together. A registry without enforceable permissions is an inventory. A permission without logs is difficult to audit. A log without an incident owner is evidence without response.

The finish state is simple to describe: the organization can identify every active agent, explain what it is allowed to do, reconstruct a material action, stop it, recover the business process, and remove its access when the agent is no longer needed.

Google Cloud Next 2026 presented one vendor architecture for these jobs. NIST, OWASP, and Cloud Security Alliance provide the vendor-neutral risk and security frame.

flowchart TB
    A["Register the agent and owner"] --> B["Assign identity and purpose"]
    B --> C["Authorize context and tools"]
    C --> D["Set action and approval limits"]
    D --> E["Evaluate before release"]
    E --> F["Observe production behavior"]
    F --> G{"Material failure?"}
    G -->|"No"| H["Review drift and renewal"]
    H --> F
    G -->|"Yes"| I["Contain, recover, and investigate"]
    I --> J["Correct, restrict, or retire"]
    J --> E

Agent governance is a lifecycle with a recovery loop.

One agent and one thousand agents are different systems

A pilot team can remember who built an agent. It can inspect a small prompt, watch a demonstration, and message the creator when something breaks.

At scale, memory becomes unreliable. Agents are copied, modified, embedded in other workflows, connected to new tools, and left running after their original owner changes roles. One agent may call another. A shared credential may hide which component performed an action. A prompt update may change behavior without a release record.

The number of agents is not the only scaling variable. Autonomy, action cost, data sensitivity, duration, and dependency depth matter more. Ten agents that only summarize public text may present less operational risk than one agent allowed to modify payroll or customer access.

Governance should therefore classify agents by capability and consequence, not apply one approval ceremony to everything.

Step 1: Create a registry that can answer operational questions

The registry is the control system's map. Cloud Security Alliance recommends an inventory with the agent's identity, owner, permissions, tools, data, and business process. Google Agent Registry is one implementation of this pattern.

The useful record goes beyond a name and description. It should identify the business outcome, accountable owner, technical maintainer, current version, model and runtime dependencies, data domains, tool endpoints, credential path, approval policy, evaluation suite, deployment environment, downstream agents, incident route, renewal date, and retirement status.

An agent should not become production-active until this record exists. Discovery and approval are separate fields. A catalog may list an experimental agent while policy prevents production use.

The registry also needs a way to find unregistered agents. Cloud logs, identity systems, gateways, code repositories, and workflow platforms can reveal active components that the catalog missed.

Step 2: Give the agent its own identity

Do not let a fleet of agents operate through one broad service account when the platform can issue distinct identities.

A unique machine identity makes policy and attribution more precise. The organization can restrict one agent without breaking unrelated work. Logs can identify the actor. Credentials can expire or be revoked when the agent is retired.

Identity does not remove the human chain of accountability. The registry should still connect the machine identity to a business owner and technical steward.

Use least privilege and time-bounded credentials. Authorize the smallest data and action set that completes the task. Separate development, test, and production. Require stronger controls when an agent can move money, disclose sensitive information, change access, publish externally, delete records, or trigger a physical consequence.

Google Agent Identity, AWS AgentCore Identity, and conventional cloud identity systems offer different mechanisms. The requirement is stable even when the product changes.

Step 3: Govern context as a source system

An agent's answer is shaped by the context it receives. That context can include documents, messages, database records, memory, retrieval results, instructions, and output from another agent.

Each source needs an authority label. A signed contract may control a commercial term. A CRM field may control the current account owner. An old email may provide history but not current policy. A model should not treat all retrieved text as equally authoritative.

Access also needs purpose. An agent that schedules a meeting may need calendar availability. It probably does not need the full content of every attendee's email.

Persistent memory deserves its own policy. Define who can write memory, what can be stored, how it is corrected, when it expires, how a person can inspect it, and how deletion propagates. A useful memory system that retains stale or sensitive information without control becomes a new source of risk.

Dalton's E113 trial offers the benign version of this problem. Gemini Enterprise found fragments in his Workspace and suggested relevant agents, but much of his useful work lived elsewhere or was intentionally private. The system could not infer the missing operating context.

Step 4: Put tools behind an enforceable gateway

Tools turn an agent's interpretation into an action. They need stricter controls than a text response.

A gateway or equivalent policy layer should authenticate the agent, authorize the requested operation, validate inputs, enforce rate and spend limits, apply data rules, record the call, and return a controlled result. High-consequence tools should support dry runs, idempotency, explicit approvals, reversible operations, or compensating actions.

Model Context Protocol can make tools easier to discover and invoke. It does not decide whether a tool is safe for this agent, user, record, and moment. That remains an implementation responsibility.

Treat external content as untrusted input. A webpage, email, document, issue, or repository instruction can contain text designed to redirect an agent. Google's security research on persistent instruction files shows why a familiar configuration file can become part of the attack surface.

Step 5: Separate interpretation from authority

An agent may be good at interpreting an ambiguous request without being authorized to complete the final action.

Define action classes by consequence and reversibility. A low-risk recommendation may be released automatically. A reversible internal update may proceed under a threshold. A customer message may require review. A payment, access change, public publication, legal filing, or destructive operation may require explicit approval through a separate control channel.

Approval should not be a vague button. The reviewer needs the proposed action, relevant evidence, uncertainty, policy basis, affected systems, and reversal path. Otherwise the system transfers responsibility without supplying enough information to exercise it.

Keep deterministic controls outside the model where possible. A model can recommend a refund amount. A policy service can enforce the maximum. An agent can draft a change. A deployment system can require tests and an authorized release.

Step 6: Evaluate the workflow, not only the model

Model benchmarks do not tell you whether an agent can complete your process safely.

Build evaluations from representative tasks, edge cases, prohibited actions, stale context, conflicting sources, tool failures, delayed responses, prompt injection, and downstream-agent errors. Measure task completion, correctness, policy compliance, evidence quality, latency, cost, recovery, and the rate of unnecessary escalation.

Version the evaluation set and the agent together. Re-run it when the model, prompt, tool, data contract, permission, memory policy, dependency, or business rule changes.

NIST's framework emphasizes measurement as part of risk management. Google Agent Platform includes evaluation services. Neither replaces the organization's responsibility to define what success and unacceptable failure mean for the actual workflow.

Step 7: Make production behavior reconstructable

Observability must answer more than whether the service was online.

For a material action, the team should be able to reconstruct the requesting user or system, agent identity and version, relevant policy, approved context references, tool calls, approvals, output, downstream effects, timestamps, cost, and final state. Sensitive content should be protected, but hiding every input makes investigation impossible.

Trace relationships across agents. If an intake agent delegates to a compliance agent that calls a customer system, a single incident view should connect the chain.

Monitor drift as well as failure. A rising refusal rate, longer latency, increased cost, repeated human correction, or a shift in tool selection can signal degradation before an obvious incident.

Step 8: Design recovery before autonomy

A production agent needs a failure contract.

Define how the system stops new work, revokes access, preserves evidence, isolates a session, reverses or compensates for completed actions, returns the business process to a person, communicates impact, corrects data, and determines whether the agent can resume.

A kill switch is useful only if people know who can use it and what happens next. Stopping an agent may leave customer requests, partial transactions, or external commitments in an uncertain state.

Run recovery exercises. Simulate a bad instruction, compromised credential, unavailable dependency, incorrect bulk action, and downstream agent that returns plausible but false information. Time the containment and restoration.

Step 9: Renew or retire deliberately

Agents should not receive permanent production status by default.

Set a renewal date based on risk. Review owner, purpose, value, permissions, dependencies, evaluation performance, incidents, data sources, cost, and user adoption. Remove duplicate or unused agents.

Retirement means more than hiding the agent from a gallery. Revoke identities and credentials. Remove schedules and triggers. Disable endpoints. preserve required records. Transfer unfinished work. Update downstream dependencies. Delete memory and retained data according to policy.

An unmanaged retired agent is still an active risk if its access remains valid.

A compact control inventory

This inventory can begin in a spreadsheet or database, but it should eventually connect to the systems that issue identities, deploy agents, route tools, and record activity. Manual governance does not scale if the production system can bypass it.

The operating principle

Thousands of agents do not become manageable because every agent is individually reliable. They become manageable because the organization can bound, observe, and recover the system.

The platform should make the safe path easier: register before deployment, assign identity automatically, restrict tools by policy, preserve traces, run evaluations on change, and remove access at retirement.

Google Cloud Next showed many of these controls inside one stack. Other platforms offer related patterns, and open protocols can connect them. The organization's governance model still has to survive a change in vendor, model, team, or workflow.

That is the standard worth designing for.

Continue the series

The Agent Platform explainer maps Google's specific components to this lifecycle. The RPA comparison shows where deterministic controls can reduce unnecessary autonomy. Episode 65 explains how MCP and A2A connections change the security and delegation surface.

Sources and disclosure

The vendor-neutral frame uses NIST AI RMF, the NIST Generative AI Profile, OWASP Agentic AI threats and mitigations, the Cloud Security Alliance agent-governance note, and its nonhuman identity whitepaper. Google's Agent Platform overview supplies one current product architecture. Google's instruction-file security analysis supports the persistent-instruction boundary.

The framework is Venture Step synthesis, not a certification or guarantee. AI assisted with research organization and drafting; source boundaries and final editorial decisions remain Dalton Anderson's.