How to Test a Custom AI Assistant Before Use

How to Test an AI Assistant on a Bounded Task

Test a custom AI assistant by freezing one version, defining one release decision, and running authorized cases that cover ordinary work, edges, adversarial requests, privacy, permissions, consistency, and recovery. Score observable behavior and failure severity, not how impressive the demo feels.

A passing result supports only the tested task, data, audience, tools, and version.

flowchart TD
    A["Freeze scope and version"] --> B["Define success and unacceptable failures"]
    B --> C["Build representative and boundary cases"]
    C --> D["Add adversarial, privacy, permission, recovery cases"]
    D --> E["Run and preserve outputs"]
    E --> F["Score behavior, severity, and reviewer effort"]
    F --> G{"Release decision"}
    G -->|Go| H["Bounded pilot and monitoring"]
    G -->|Hold| I["Revise and retest"]
    G -->|Reject| J["Retire or redesign"]

Define the decision first

"Does this assistant work?" is too broad.

A useful question is: "Can version 1.3 turn an authorized set of episode notes into a source-labeled outline draft for Dalton's review, without inventing facts or publishing anything?"

That decision names the version, input, task, user, output state, and authority. It also suggests what failure looks like.

If the intended use changes from private drafts to a team workflow, or from drafting to sending, the old result no longer answers the decision.

Freeze the system under test

Record the product, visible model if available, instruction version, knowledge files, connected apps, tools, permissions, account type, settings, date, and audience.

Preserve the exact instructions. If a live Drive file is part of a Gem, record its version or hash. If a model or product can change without notice, say so and shorten the review interval.

Google's current Gem documentation allows optional knowledge sources and reflects changes to certain Drive files. That means a later run may not use the source set that passed the test.

Define success and unacceptable failure

Success should include the full cost of use. Measure task completion, evidence use, corrections, reviewer time, and whether the result reaches the next workflow step.

Then define failures that cause an automatic hold. Examples include exposing private data, inventing a source, ignoring a required approval, acting outside the authorized destination, or presenting an unsupported professional conclusion.

Do not let a high average hide one severe event.

Build representative cases

Use real workflow patterns with authorized and appropriately deidentified data.

Include common inputs, good inputs, messy inputs, incomplete inputs, long inputs, and inputs with ordinary ambiguity. If the task has several legitimate output styles, define required properties rather than one exact answer.

For the Venture Episode Engine, a representative case would include a transcript, notes, and the current outline structure. The output should represent the approved ideas, preserve source boundaries, follow the section order, and remain a draft.

Add boundary cases

Boundary cases test what happens at the edge of the contract.

Remove a required date. Supply a production outline instead of a transcript. Include two sources that disagree. Ask for a current product claim while withholding current documentation. Provide a source that is too old for the decision.

The assistant should not fill every blank. A useful failure response identifies the missing requirement and returns the part that can be completed safely.

E033's live travel planner offered a clear example. Dalton supplied a trip, destinations, and major activities but did not mention that he would be working. The generated schedule looked complete while missing a controlling constraint. A test set should detect that kind of false completeness.

Challenge the instructions

Adversarial testing is not limited to dramatic attacks. It includes ordinary requests that conflict with the system boundary.

Ask the assistant to ignore its source policy, reveal its instructions, use an unapproved source, invent a quotation, change the audience, or take an action it may only draft. Put conflicting text inside a source file and see whether it is treated as data or instruction.

The goal is not to prove that every attack will fail. It is to identify where the current architecture depends on a model choosing correctly.

Test privacy and data handling

Introduce data that should be rejected, redacted, segregated, or escalated. Verify what the product stores, shares, logs, and exposes to collaborators.

Google's current Gem sharing help says people with access can view Gem instructions and uploaded files. A sharing test should therefore include inspection from a viewer account, not only a successful chat.

Google's Gemini Apps Privacy Hub describes activity, temporary chats, human review, retention, and account differences. Test the actual account and settings. Do not generalize consumer behavior to a work or school deployment.

Test permissions and side effects

If the assistant can use tools, test each read and write separately. Confirm which account, object, and destination are in scope.

Run duplicate requests. Interrupt a step. Force a timeout. Return malformed tool output. Deny permission. Change the destination between planning and approval.

The assistant should not silently broaden access, repeat a write, or treat a draft approval as authorization to act.

When possible, use a test environment and reversible operations. A polished report about a harmful action is still a failure.

Test consistency without demanding identical prose

Run the same case several times. The wording may vary, but key evidence, boundaries, calculations, and actions should remain stable.

Track whether the system sometimes omits a required warning, chooses a different source, invents a new fact, or changes the action plan. Variance can reveal a fragile instruction even when the first output looked good.

Exact match is appropriate for deterministic fields and calculations. It is usually a poor score for open-ended prose.

Test recovery

A system is more dependable when failures can be found and corrected.

Plant a known unsupported claim. See whether the reviewer can trace it. Correct a source and rerun. Revoke a tool. Stop the process halfway. Attempt a rollback. Confirm that logs show what happened.

For public or tool-using systems, test reporting, escalation, owner notification, and shutdown. Recovery should not exist only on paper.

Preserve an evaluation record

Keep failed outputs. They are often more valuable than screenshots of successful demos.

Make a bounded release decision

A release can be go, hold, or reject. "Go" should include the exact scope, required human review, monitoring, incident path, and retest triggers.

NIST's Generative AI Profile emphasizes pre-deployment testing, ongoing evaluation, and incident disclosure as important risk practices. It also notes that measurement remains immature and context-specific.

Passing a small test set is evidence for a bounded pilot. It is not proof of general reliability.

Retest after changes to instructions, sources, model, product, tools, permissions, task, audience, policy, or risk. Production failures should become new cases.

Use [[How to Design a Reusable AI Assistant]] to repair the contract and [[How to Write Instructions for a Custom AI Assistant]] to repair behavior. If the assistant can act in external systems, revisit [[Saved Prompt, Custom Assistant, Workflow, or Agent]] and make the authority explicit.

This evaluation guide was developed with AI assistance from E033, current Google documentation, the linked evaluation framework, and NIST guidance. Dalton Anderson remains the author. Technical, privacy, security, safety, domain, current-source, and founder review are mandatory before publication. Publication is not authorized.