How to Evaluate an AI Workflow Before Activation

How to Evaluate an AI Workflow Before You Turn It On

An AI workflow is ready only when its purpose, trigger, inputs, model decisions, permissions, actions, recipients, failure modes, tests, logs, rollback, limits, owner, review date, and retirement condition are explicit.

A successful builder demo is evidence that one path ran once. Activation means authorizing the system to encounter new data and repeat its actions without the builder open.

Google Workspace Studio makes the difference concrete. Google's test-run guidance says a test uses real data and takes real actions. It can send messages, change files, or create meetings. E097 recorded a successful attachment save and an accidental exposure of private information during the same inbox experiment.

flowchart TD
    A["Define the job and harm boundary"] --> B["Map data, permissions, decisions, and actions"]
    B --> C["Design positive, negative, failure, and repeat tests"]
    C --> D["Inspect evidence and unresolved risk"]
    D --> E{"Can high-impact errors be prevented or held?"}
    E -->|No| F["Do not activate"]
    E -->|Yes| G["Assign owner, monitoring, rollback, and review"]
    G --> H["Activate narrowly"]
    H --> I["Observe, revise, or retire"]

The activation record

Create one record before the live run. It should be short enough to use and specific enough that another person can operate the workflow.

The record is not a security certification. It is a decision surface. If a material field cannot be answered, the workflow is not ready for the authority it currently has.

1. Define one job and its harm boundary

Write the workflow's purpose without naming the product. "Handle important emails" is too broad. "Copy PDF attachments from three approved senders into a private review folder" is testable.

Describe the finish state and the harm boundary. For an attachment flow, success may be one expected file in one destination. Harm may include copying a health document, customer record, or malicious file into an unmanaged repository.

The NIST AI RMF Core includes defining the tasks, context, affected people, human roles, and ongoing monitoring. NIST presents the framework as voluntary and use-case-neutral, not as a checklist that certifies a system.

2. Map the trigger before the model

Every unnecessary event expands the workflow's exposure. Record whether the trigger is a schedule, inbound message, file change, form submission, user request, or another system.

Write what should not trigger the workflow. Test those exclusions. A flow that handles attachments should encounter a message without an attachment, an unapproved sender, and a subject that almost matches but does not.

Narrowing the trigger also protects capacity. Google's Workspace Studio limits page recommends more specific starters when unnecessary runs consume the daily limit.

3. Trace every data element

Follow data from the starter through each variable, prompt, decision, action, recipient, log, destination, and outside service. Name the classification and owner of each source.

Do not stop at "the flow can access Drive." Ask which files the initiating user can access, which ones Gemini may use, what enters the prompt, what appears in the output, and who receives it.

If a third-party connector appears, draw a new service boundary. Google's integration guidance warns that variables can carry Google Account data into the connected service. Its permissions, retention, deletion, and contract need their own review.

4. Separate fixed rules from model judgment

A fixed rule such as "subject contains the test token" is easier to verify than a model judgment such as "this email seems urgent." Use deterministic checks when the condition can be expressed reliably.

When a model is necessary, record the prompt, allowed context, output shape, expected uncertainty, and downstream action. Test ambiguous and adversarial inputs, not only clean examples.

Google's conditional-step guidance distinguishes Check if, which evaluates declared conditions, from Gemini-assisted decisions for subjective criteria. The product choice should follow the task, not the excitement of adding AI.

5. Put approval before the costly action

Human review belongs before an action that moves money, deletes data, changes permissions, publishes content, makes a contractual commitment, reaches an uncontrolled recipient, or influences a regulated decision.

The reviewer needs the source, proposed action, reason, uncertainty, and consequence. An approval prompt that says only "continue?" transfers responsibility without supplying judgment.

Low-risk, reversible, observable actions may not need an approval on every run. The objective is not maximum friction. It is to keep the cost of a wrong action inside a boundary the organization has accepted.

6. Test more than the happy path

Run a controlled positive case, several nonmatching cases, malformed and empty values, a duplicate, a repeated event, revoked access, an unavailable destination, and the largest realistic input.

For every test, write the expected starter behavior, decision, action, visible evidence, and cleanup. Use synthetic or copied data and self-only recipients.

Google's test guidance recommends that approach. Its troubleshooting documentation also identifies failures caused by invalid data, configuration changes, lost access, administrator restrictions, limits, security measures, unavailable steps, and bugs.

A green run is incomplete evidence if no one checked the destination and the log. A red run is also incomplete evidence if an earlier step already sent or changed something.

7. Prove observability and rollback

Identify where success, delay, and failure appear. Name who reviews that surface and how quickly. If the product only reports errors inside an activity page, decide who will visit it.

Practice turning the workflow off. Revoke one permission in a test environment if safe, and confirm how the failure appears. Reverse a file write or remove a synthetic record. If an action cannot be reversed, the workflow needs stronger prevention and approval before it happens.

The NIST Generative AI Profile emphasizes continuous monitoring, third-party risk, incident response, fallback technologies, and named responsibilities. Those ideas apply even when the visible automation seems small.

8. Check limits, change, and ownership

Record product quotas, frequency, expected volume, service dependencies, and administrator controls. Decide what happens when a limit is reached or a step disappears.

Name the workflow owner, operational reviewer, approval authority, and escalation path. One person may hold several roles in a small team, but the roles should still be explicit.

Set a review date and a retirement condition. A flow should be reconsidered when its data changes, its error rate rises, the platform changes, the owner leaves, the task becomes regulated, or the manual fallback is no longer viable.

Choose the narrowest safe release

Activation should begin with the smallest live scope that can produce useful evidence. Limit senders, recipients, destinations, frequency, actions, or the period of operation. Watch the first runs.

If high-impact errors remain possible and cannot be detected before action, hold the workflow. Move the action behind approval, reduce its authority, replace the model decision with a fixed rule, or return the task to a person.

For task selection, continue with Should This Task Be a Workflow, an Agent, or Manual?. For a concrete low-risk build, use How to Save Gmail Attachments to Drive with Workspace Studio.

E120's [[How to Manage an AI Coworker|AI coworker management guide]] extends the same principle to a system with broader discretion. The outcome, authority boundary, and proof of completion become more important as the system receives more freedom over the path.

This guide was developed from NIST materials, current Google documentation, and the preserved E097 experiment. AI assistance was used for research organization, drafting, and validation. Regulated and high-consequence workflows require additional domain review.