Permission Before Autonomy: A Family Office Charter for AI Agents

Start with the action

A system that summarizes an investment memorandum and a system that transfers money may share a conversational interface, but they do not share the same consequences. Treating both as an AI adoption initiative hides the most important governance decision: which actions can occur without another authorized person intervening?

A family office should assign permissions to tasks, not to a brand name or an impressive demonstration. This article proposes a task-level authority ladder. It begins with reading and drafting, progresses to reversible internal changes, and treats external commitments and sensitive transactions as distinct decisions requiring stronger controls. This is an editorial design, not a claim that autonomous systems have been validated for a particular office.

Translate a framework into boundaries

NIST's voluntary AI Risk Management Framework organizes work around governing, mapping, measuring and managing risk.[1] Its generative-AI profile discusses risks including confidently incorrect content and fabricated supporting citations.[2] Those documents support an important distinction: a plausible explanation is not itself evidence that a system acted correctly.

The office's charter should therefore define a permitted task, an authorized data set, allowed tools, forbidden actions, a responsible owner and a recovery method. A procurement assistant might compare approved invoices against contracts while being unable to alter bank details or release payment. An investment assistant might retrieve cited passages while being unable to approve a subscription or send confidential materials to an outside party.

Test the boundary, not only the happy path

Before access is granted, the operations lead assembles examples of ordinary work, ambiguous instructions, missing information and conflicting documents. The security lead includes attempts to induce unauthorized actions through text embedded in a document. The system must demonstrate that it can refuse or escalate tasks outside its authority. A model's broad benchmark score does not replace testing the actual workflow and permissions.

Keep a small, versioned evaluation set containing expected source passages, permitted outcomes and unacceptable actions. Measure unsupported factual statements, missed escalations, unauthorized tool attempts and human correction time. Evaluate the whole workflow: retrieval, prompts, model, tools, human review and final action. Changing one component can change the system's behavior even when the interface looks unchanged.

A hypothetical invoice assistant

Suppose an office receives an invoice bearing new payment instructions and an urgent request from someone claiming to represent a familiar vendor. The assistant can identify the mismatch, locate the existing contract and prepare a verification task. It cannot decide that familiarity with the supplier makes the change safe. A designated staff member verifies the change through an established independent channel before an authorized payment process begins.

In this example, success is not the number of invoices processed without interruption. It is accurate handling of ordinary invoices and reliable escalation of the exceptional one. The office retains the source documents, the system's proposed action, the verification record and the human decision. No actual incident or measured reduction in fraud is asserted.

Give rollback an owner

The chief operating officer is accountable for the charter. Each workflow has a business owner, and the person who approves broader authority should be able to inspect the test evidence. A small office can assign several roles to one person, but it should not let a vendor's enthusiasm substitute for its own acceptance decision. Access should be limited to what the task needs, and logs should be protected as sensitive records.

Stop the workflow immediately after an unauthorized external action, a confirmed disclosure of protected information or a failure of a critical escalation rule. Suspend expansion of authority when evaluation coverage is incomplete. Record how to revoke access, restore prior records, notify responsible internal staff and continue the work manually. The ability to turn a system off is meaningful only if the office can still operate afterward.

Measure capacity without surrendering judgment

Track total time saved after review and correction, rather than counting only the time used to generate a draft. Also record error severity, unresolved exceptions and tasks that staff now avoid because they trust the system too readily. Revisit permissions after vendor updates, new data sources or changes in business purpose.

The charter does not promise safety or remove professional duties. Its value is to make authority explicit and evidence visible. An office may discover that a narrowly bounded assistant is more useful than a broadly autonomous agent because people can understand, test and trust the limits of its work.

Editorial method, evidence and limitations

This original AI-assisted editorial analysis proposes a decision framework. It is not a new empirical study, systematic review, or independently human-peer-reviewed article. Cited institutional sources inform the framing; the workflow, illustrative scenarios and suggested measures are editorial proposals. No source author, conference organizer or sponsor is represented as endorsing the article.

AI-assisted editorial workflow proposal, not a security certification or validation of any model. Controls require task-specific testing and ongoing maintenance.

Open the cited sources

1. [1] NIST, AI Risk Management Framework ↗
2. [2] Autio et al. (2024), Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 ↗