How to Evaluate an AI Tutor in Your Organization

How to Evaluate an AI Tutor for a School or Workplace

Evaluate an AI tutor through one low-risk learning or work task. Approve the sources and account first, measure the existing baseline, audit claims and citations, test accessibility and oversight, exercise the incident and exit paths, then decide whether to stop, revise, or expand.

Do not measure success by the novelty or volume of generated output.

flowchart TD
    A["Approved low-risk job"] --> B["Source, account, rights, and data review"]
    B --> C["Baseline without the tool"]
    C --> D["Limited participants and oversight"]
    D --> E["Claim audits and independent outcome"]
    E --> F["Accessibility, integrity, and incident review"]
    F --> G{"Decision"}
    G -->|Stop| H["Export evidence and delete approved data"]
    G -->|Revise| B
    G -->|Expand| I["New scope and fresh approval"]

Define the job before the platform

A school and a workplace do not share one legal, pedagogical, labor, or governance context. The pilot method can be common while the approvals remain different.

Choose one job with a visible result.

A school might test whether students can explain a defined concept from approved readings after using a source-grounded study workflow. A workplace might test whether employees can locate and explain a current low-risk procedure.

Avoid discipline, grading, diagnosis, eligibility, hiring, performance management, safety-critical action, legal interpretation, financial advice, or another high-impact use.

Write who owns the outcome and who has authority to approve the pilot.

Define who may participate

Record the participant group, age or employment context, inclusion and exclusion criteria, consent or notice, support, alternatives, and withdrawal path.

Participation should not force someone to disclose disability, personal information, academic difficulty, employment concerns, or confidential work merely to receive equivalent access.

Provide a non-AI path. A pilot cannot show value if people must accept the tool to complete the required task.

In a school, the educator and institution own the learning and academic-integrity rules. In a workplace, the process owner and organization own the work standard and labor context.

Approve the source pack

Use [[How to Build a High-Quality Source Pack for an AI Tutor]] to record every source's identity, version, authority, scope, rights, data class, conflicts, and gaps.

Do not upload student records, personnel files, customer information, unpublished work, confidential policy, licensed course content, or regulated data without explicit authority and the required controls.

Test the pack with answerable, disputed, and unanswerable questions.

Assign a source owner. When a policy or reading changes, the owner needs to know which notebook and assessments depend on it.

Verify the exact account and terms

Record product, edition, subscription, model, region, administrator, sign-in method, connected services, and applicable terms.

Google's current NotebookLM privacy and terms page distinguishes consumer, work, school, and cloud use. Current vendor documentation says Workspace and Education accounts receive different data treatment from consumer feedback flows.

That is a first-party statement, not proof that the institution configured the service correctly.

Verify administrative controls, sharing, public links, exports, feedback, retention, deletion, activity, logging, support, incident reporting, and account recovery in the actual tenant.

Treat sharing as access

Do not assume a cleaner interface hides underlying material.

Google's current public-notebook documentation says viewers may still reach sources or artifacts even when a chat-focused view hides them.

Test every participant role with representative accounts. Confirm what an owner, editor, viewer, external user, former participant, and public visitor can see and do.

Remove public sharing unless the sources and outputs are explicitly approved for that audience.

Measure the baseline

Before introducing the tool, measure the present workflow.

For learning, use a comparable explanation, retrieval task, or application problem. For work, measure the ability to locate the current procedure, explain it, and perform the approved task.

Record time, accuracy, uncertainty, escalation, and support needs where appropriate.

The baseline does not need to be elaborate. It needs to be comparable and honest.

Run a limited cohort

Use the smallest group and duration that can answer the pilot question.

Provide clear instructions on permitted sources, permitted prompts, attribution, academic integrity, confidentiality, prohibited uses, error reporting, and human support.

Do not ask participants to discover the boundaries through mistakes.

Keep an accountable educator or process owner available. The assistant should not become the only path to clarification.

Audit source fidelity

Sample answers across easy, ambiguous, conflicted, and unanswerable questions.

Use [[How to Verify an AI Answer Against Its Citations]] to check identity, passage, support, context, inference, coverage, date, and authority.

Track unsupported claims, missing qualifications, stale sources, citation failures, refusals, and confident answers to questions outside the pack.

The NIST Generative AI Profile provides a broader risk frame for confabulation, privacy, information integrity, human configuration, and governance.

Measure an independent outcome

After using the tool, ask participants to demonstrate the target capability without it.

A student might explain or apply the concept with approved assessment conditions. An employee might complete the task using the official process and know when to escalate.

Do not count chat volume, notebook opens, generated audio minutes, summaries, or quizzes as the primary outcome.

Compare the result with the baseline. Record uncertainty and adverse effects, not only average improvement.

Test accessibility and alternatives

Test with people who represent the intended population, including disabled participants where appropriate and voluntary.

Review keyboard access, screen-reader behavior, captions or transcripts, audio controls, color and visual dependence, cognitive load, language, mobile access, error messages, time limits, and recovery.

A generated audio option does not establish accessibility. A text option does not establish equivalence.

Google's current education offering describes vendor protections and availability. Institutional accessibility review and participant evidence still control the decision.

Define integrity and attribution

State when AI assistance is permitted, how it must be disclosed, which work must be independent, and how citations should be preserved.

An education pilot should not blur tutoring with assessment. A workplace pilot should not turn generated answers into undocumented policy.

Preserve the original source link and decision owner. Do not let a generated note become a shadow system of record.

Exercise the incident path

Before expansion, run a tabletop exercise.

Test what happens when a notebook exposes an unauthorized source, produces a harmful answer, cites a superseded rule, shares material too broadly, loses access to a source, or cannot delete participant data as expected.

Record who receives the report, who can disable access, how affected people are notified, how the source and output are preserved for review, and how a correction reaches prior users.

The U.S. Department of Education's student privacy and education technology resources provide institutional questions for online educational services. They are not a product approval or legal conclusion.

Decide with evidence

Stop when the use case is unsuitable, sources are unauthorized, participants cannot withdraw, access cannot be controlled, incidents cannot be handled, or outcomes cannot be measured.

Revise when the job remains useful but the source pack, instructions, configuration, assessment, or support needs work.

Expand only for the named task after the required education, accessibility, academic-integrity, privacy, security, copyright, employment, legal, procurement, product, source, and leadership reviews.

Expansion creates a new scope. It requires fresh evidence and approval.

For a related organization-level software pilot, read [[How to Evaluate an AI Coding Tool for a Development Team]].

This guide was developed with AI assistance from the immutable E035 transcript, current Google NotebookLM and Education records, NIST's Generative AI Profile, U.S. Department of Education privacy material, and the linked institutional pilot framework. Dalton Anderson remains the author. Education, accessibility, academic integrity, privacy, security, copyright, employment, legal, procurement, product, current-source, and founder review are mandatory before publication. Publication is not authorized.