AI governance for autonomous systems: from policies to runtime control

For most of the last decade, AI governance was designed around a relatively passive model of artificial intelligence: a model received an input, produced an output, and a person or another software system decided what happened next. Governance could therefore concentrate on training data, output quality, fairness, privacy, explainability, and approval for a use case.

Autonomous and agentic AI changes the problem. An agent can interpret a goal, plan, retrieve information, choose tools, authenticate to systems, execute actions, observe results, revise its plan, coordinate with other agents, and continue until it decides the task is complete. The governance question is no longer only “Can we trust what the AI says?” It is also “What may this system do, under whose authority, with which resources, under what conditions, and how do we stop it?”

Why traditional AI governance is no longer enough

A model can be accurate and still be unsafe when it is connected to a payment system, a customer record, a deployment pipeline, or a privileged internal API. The risk is created by the combination of model behavior, tools, data, credentials, memory, orchestration, and the environment in which actions occur.

• Information risk: an answer is wrong, biased, stale, private, or unsupported.
• Action risk: the system sends, edits, purchases, deploys, deletes, escalates, or grants access incorrectly.
• Control risk: nobody can reliably explain which identity acted, which policy applied, what evidence was used, or how to stop the workflow.
• Emergent risk: a chain of individually permitted steps produces an outcome that no single component was designed to authorize.

This is why agent governance has to cover identity, authorization, auditing, non-repudiation, prompt-injection resistance, tool boundaries, memory, delegation, and recovery. NIST is explicitly studying software and AI-agent identity and authorization; OWASP has created an Agentic Security Initiative; and major enterprise platforms are adding runtime policy controls for agent-tool interactions.

Govern the system, not only the model

Runtime governance becomes architecture

Design-time controls define what should be possible. Runtime controls decide what is allowed now. The runtime layer should receive the proposed action, the agent identity, user or service authority, resource, data classification, task context, risk signals, and current policy. It should return a decision that is enforceable by the tool gateway or application—not merely a recommendation for the model.

• Design time: classify impact, define owners, identify assets, threat-model misuse, write policies, test guardrails, and establish approval paths.
• Runtime: authenticate the caller, resolve authority, evaluate policy, constrain arguments, require approval when needed, execute through a gateway, and emit evidence.
• After execution: verify the result, record the outcome, update risk signals, and trigger recovery or review if the action deviated.

Identity, authorization, and delegated authority

Authentication answers “Who are you?” Authorization answers “What are you allowed to do?” An authenticated agent can still be unauthorized to access a record, call a tool, change a resource, or act on behalf of a user. Treating the two as interchangeable is one of the most common design errors in agent systems.

• Give every agent, workflow, and delegated sub-agent a distinct, traceable identity.
• Bind credentials to a task, environment, audience, expiry, and least-privilege scope.
• Represent user authority and agent authority separately so delegation is visible.
• Log the policy version, identity chain, resource, decision, and evidence used for each consequential action.
• Make revocation immediate enough to matter during an incident.

Policy-as-code turns intent into enforcement

Natural-language principles such as “agents must protect customer data” are important, but they are not executable. A policy needs a subject, action, resource, context, conditions, decision, and enforcement point. Policy-as-code makes those decisions reviewable, testable, versioned, and consistent across services.

permit(principal, action, resource) when
  principal.agent_type == "support-triage"
  && action == "ticket.update"
  && resource.owner == principal.tenant
  && resource.classification <= "internal"
  && context.approval == true;

forbid(principal, action, resource) when
  action in ["payment.send", "credential.rotate"]
  && context.human_approval != true;

The syntax is less important than the separation of concerns: the model proposes, the policy engine evaluates, and the tool gateway enforces. A model should not be the final authority over the policy that governs its own action.

Autonomy should be earned

• Inform: summarize, classify, or recommend; no external side effects.
• Assist: prepare a draft or action plan; a person confirms before execution.
• Bounded execution: act within a narrow tool set, data scope, budget, and time window.
• High autonomy: coordinate multiple steps or agents; requires stronger authorization, monitoring, evaluation, and recovery.

Autonomy is not a single slider. It is a vector across action sensitivity, reversibility, data access, tool breadth, delegation depth, time horizon, and human-review latency. A low-risk read-only research agent may be highly autonomous in searching while remaining completely unable to write to a system.

Human oversight should be risk-based

“Human in the loop” is not a sufficient control description. A human may be asked to approve too many low-value actions, approve too quickly to review meaningfully, or be absent when a high-impact decision occurs. Oversight should be designed around consequence and uncertainty.

• Require approval for irreversible, external, financial, legal, safety-sensitive, or high-privacy actions.
• Use sampling and retrospective review for low-risk, reversible actions.
• Escalate when evidence conflicts, confidence is low, policy conditions are missing, or the agent exceeds its budget.
• Show the reviewer the proposed action, evidence, policy reason, expected impact, and rollback path.
• Measure whether review changes outcomes instead of counting approval clicks.

Tools, memory, and delegation are governance surfaces

Every tool is an authority boundary. Tool descriptions should state the action, input schema, resource scope, side effects, data handling, idempotency, error behavior, and situations where the tool must not be used. A vague tool such as `updaterecord()` makes safe authorization difficult; a scoped tool such as `updatesupportticketstatus(ticket_id, status)` is easier to validate and audit.

• Tool governance: allowlists, typed arguments, schema validation, policy checks, rate limits, dry runs, approval gates, and audit events.
• Memory governance: classify memory, set retention and provenance, separate user preferences from facts, prevent untrusted content from becoming policy, and support deletion.
• Delegation governance: define which sub-agents may be created, what authority they receive, how long it lasts, and whether they may delegate again.
• MCP and connectors: treat every external server or connector as a supply-chain and authorization boundary with an owner, version, scope, and revocation path.

Observability and evaluation become evidence

Logs are not automatically evidence. Evidence is a trustworthy record that connects an action to an identity, policy, input, context, tool call, result, and reviewer or system decision. Preserve enough information to reconstruct what happened without collecting unnecessary sensitive content.

Containment and recovery are first-class controls

A robust system assumes that models will sometimes misinterpret instructions, tools will fail, credentials will leak, retrieval will be poisoned, and policies will be incomplete. The design question is not whether failure can be eliminated; it is whether failure can be detected, contained, explained, and recovered from.

• Deny the proposed action and preserve the request for review.
• Reduce authority to read-only or remove high-risk tools.
• Pause the workflow and require a human decision.
• Revoke credentials, sessions, delegated tokens, or connector access.
• Quarantine suspicious memory, retrieved content, or tool responses.
• Use a kill switch that is independent of the model and reachable during partial outages.
• Replay the trace in a safe environment to identify the control failure.

The AI supply chain needs an inventory

Autonomous systems are assembled from models, prompts, tools, connectors, vector stores, policies, identity providers, runtime frameworks, evaluation sets, and data. A component inventory should cover the system that actually runs, not only the model card.

• Models and providers, including versions, regions, training-data constraints, and fallback behavior.
• Prompts, policies, tool schemas, routing rules, memory stores, and retrieval indexes.
• Packages, containers, connectors, MCP servers, plugins, and deployment workflows.
• Credentials, service accounts, data stores, queues, and external systems reachable by the agent.
• Owners, risk classifications, change history, test evidence, and retirement dates.

Frameworks and tooling that fit the problem

A practical enterprise workflow

Begin with one workflow whose outcome, users, data, and authority can be described precisely. Make the first version read-only or approval-gated. Build the trace before adding autonomy. Run realistic and adversarial evaluations. Expand scope only when the evidence shows that quality, safety, cost, latency, and recovery remain inside the agreed operating envelope.

A layered technology stack

References