How to Write AI Assistant Conversation Examples

How to Write Conversation Examples That Test Behavior

Write conversation examples around real situations, not decorative personality. Each example should define the user input, relevant source or rule, expected answer properties, prohibited behavior, and consequence of failure. Keep a separate blind test set to see whether the behavior generalizes.

Examples can show a model what "good" looks like. They can also hide weak design if every test repeats the same examples.

flowchart TD
    A["Choose a real situation"] --> B["Write the user input"]
    B --> C["State expected properties"]
    C --> D["State prohibited behavior"]
    D --> E["Add ambiguity or pressure"]
    E --> F["Run as configuration example"]
    F --> G["Test with unseen variants"]

Start with the behavior

Choose one behavior that matters to the assistant's job. It might be citing an approved source, asking a clarifying question, using a required structure, protecting private data, refusing an unsafe request, or escalating to a person.

"Sound friendly" is not enough. Describe observable properties.

For example, a podcast research assistant might need to answer with the episode title, guest, timestamp, source link, and a sentence that distinguishes the guest's claim from verified background research.

The desired behavior is now reviewable.

Write the input from the user's point of view

Use language a real user would type. Include shorthand, missing context, typos, urgency, and follow-up questions where they occur in practice.

Do not make every user perfectly cooperative. A public assistant should face vague, impatient, argumentative, and manipulative phrasing in evaluation.

Avoid including sensitive real-world data in examples. Use synthetic or properly de-identified cases and verify that de-identification is adequate for the context.

OWASP's prompt injection guidance is a useful source for adversarial cases in which user or retrieved content attempts to change the assistant's intended behavior.

Define expected properties, not one magic sentence

The ideal response may vary while still meeting the standard.

Record the facts that must be present, claims that require support, tone boundaries, acceptable uncertainty, required citations, output structure, and escalation behavior.

Then record what must not happen. The response may not invent a source, expose private data, imply human identity, make an unauthorized commitment, or continue when required information is missing.

This property-based approach prevents reviewers from rewarding a response merely because it resembles a preferred paragraph.

Add the boundary case

For each routine example, add a nearby case that should behave differently.

If the assistant may summarize a public episode, test a request for an unpublished transcript. If it may explain a product's documented feature, test a request to promise a future release. If it may draft a response, test a request to send it.

Boundary pairs reveal whether the model understands the distinction or only repeats a pattern.

Cover conflicts and missing evidence

Ask what the assistant should do when two approved sources disagree, a source is stale, a document is silent, or the user asks for a certainty the evidence cannot provide.

A strong example demonstrates the correct uncertainty and next step. It should not reward a confident guess.

Google's prompt design guidance describes few-shot examples as a way to regulate formatting, phrasing, scoping, and patterns. It also emphasizes consistent example formatting and iterative testing.

The platform guidance supports using examples. It does not guarantee that an example will control every future response.

Separate configuration from evaluation

Maintain two sets.

The configuration set contains carefully chosen examples that steer behavior. The evaluation set contains unseen variations, edge cases, and adversarial cases. Reviewers should run the evaluation set against a frozen version of the model, instructions, tools, and sources.

If a weak case is added to the configuration every time the assistant fails, the team may end up teaching answers without testing the underlying rule. Keep some failures blind and vary names, order, tone, and surface details.

Score the response

Use a small rubric tied to the job. Score factual support, completion, boundary compliance, citation quality, privacy, format, escalation, and reviewer effort where relevant.

Record the prompt, model, settings, source versions, output, score, reviewer, and disagreement. A single overall grade can hide a critical failure, so preserve gate failures separately.

The NIST Generative AI Profile encourages evaluation in the context of the system and its risks. That is why the consequence of failure belongs in the example record.

Learn from the failed conversation

In episode 28, Dalton noticed that Curio returned facts as flat lists. He changed the instruction toward storytelling and tested again.

The useful part was not the exact wording. It was the transition from "I do not like this" to a visible behavior that could be revised.

A rigorous version of that move would preserve the failing input and output, define the desired properties, change one element, rerun the same case, and then test unseen variations.

For instruction design, continue with [[How to Write Instructions for a Custom AI Assistant]]. For a bounded evaluation method, use [[How to Test an AI Assistant on a Bounded Task]] from episode 33.

This guide was developed with AI assistance from E028, Google AI documentation, NIST, and the linked test specification. Dalton Anderson remains the author. Editorial, technical, safety, domain, accessibility, source, and founder review are required before publication or use. Publication is not authorized.