How to Evaluate a Mixed-Reality Product Demo

How to Evaluate a Mixed-Reality Product Demo

A mixed-reality demo proves only that the documented setup produced the documented behavior under its conditions. To judge it well, record the claim, article, environment, operator, control mode, inputs, output, latency, duration, repetition, errors, recovery, access, and later product state.

Spectacle is not a method.

flowchart LR
    A["Exact demo claim"] --> B["Prototype or shipping article"]
    B --> C["Environment and operator"]
    C --> D["Control mode and inputs"]
    D --> E["Observed output, latency, and errors"]
    E --> F["Repetition and recovery"]
    F --> G["Narrow evidence statement"]
    G --> H["Later product and independent testing"]

Freeze the exact claim

Write what the presentation is asking the audience to believe.

"The interface placed a digital calendar on a table and responded to wrist input during the demonstration" is testable.

"This replaces the smartphone" contains assumptions about price, manufacturing, software, battery, reliability, portability, accessibility, privacy, network, support, and user behavior that a demo cannot settle.

Preserve the speaker, wording, event, date, video or source, and any qualifications.

Identify the article

Determine whether the demonstrated object is a research prototype, product prototype, developer kit, preproduction unit, shipping product, simulation, or prerecorded visualization.

Meta's Orion announcement called the device a product prototype and said Orion itself would not reach consumers.

Meta's Quest 3S launch page described a product that later became available for purchase.

Those artifacts can both appear in a keynote. They support different conclusions.

Do not let production lighting turn a visualization into a hardware result.

Record the environment

Mixed-reality performance can depend on lighting, room size, surfaces, markers, calibration, occlusion, network, temperature, noise, content, and safety boundaries.

Draw the physical space. Record the device and software version, connection, companion compute, accessories, tracking setup, accounts, calibration, and preloaded content.

A controlled environment may be the correct place to demonstrate a prototype. The control belongs in the result.

Identify the operator and support

Was the operator a product expert, trained presenter, employee, selected guest, independent tester, or member of the target population?

Record training, rehearsal, prompts, allowed paths, staff assistance, remote support, resets, and whether another person controlled any part of the experience.

Human assistance does not invalidate a demo. Hidden assistance changes what the demo proves.

Determine the control mode

Mixed-reality systems can combine live device output, scripted content, prerecorded media, animation, simulation, remote control, operator selection, and postproduction editing.

Trace each visible result to its input.

Do not infer autonomy from natural presentation.

Measure the output

Record what the user saw or heard, from whose perspective, and with what capture method.

An audience camera may not reproduce the wearer's display. A compositor may create a spectator view. A headset capture can omit optical artifacts or physical discomfort.

Measure latency, alignment, stability, field of view, occlusion, readability, audio, tracking loss, error messages, and recovery according to the claim.

Do not substitute vendor specifications for observed performance. Meta's current Quest comparison page can establish current first-party values for named devices. A task trial still needs its own record.

Preserve failures and repetitions

Ask how many trials occurred, which one was shown, how often the task failed, whether the system reset, and what conditions changed.

A single success can demonstrate possibility under the setup. Repeated trials with representative users provide stronger evidence about reliability.

Publish the failures that define the boundary. They often teach more than the polished run.

Test recovery

Trigger tracking loss, bad lighting, incorrect input, network loss, low battery, application crash, calibration drift, boundary violation, and user confusion.

Record whether the system recognizes failure, preserves physical awareness, offers a safe stop, explains the state, and restores the task.

Safety cannot be inferred from a demo that never leaves its happy path.

Separate prototype and product questions

A prototype can validate integration and produce design learning while leaving manufacturing, cost, durability, repair, support, privacy, accessibility, and market fit unresolved.

A shipping product answers availability. It does not automatically answer quality or suitability.

The NIST AI Risk Management Framework provides a useful structure when AI perception or assistants affect the experience: map the use and affected people, measure behavior and uncertainty, manage risk, and assign governance.

Write the narrow result first

The public statement should include the article, task, environment, operator, support, control mode, result, failures, and remaining questions.

A useful formulation is: "Meta demonstrated selected Orion interactions on a product prototype for controlled audiences; the reviewed source did not establish consumer price, production scale, durability, or availability."

That sentence respects both the engineering result and the evidence gap.

Use [[How to Evaluate an Engineering Demonstration]] for a broader product-neutral method. Use [[What Meta Orion Proved and Did Not Prove]] for the prototype case.

This guide was developed with AI assistance from the immutable E036 transcript, Meta's Orion and Quest records, current Quest comparison material, NIST AI RMF, and the linked demonstration framework. Dalton Anderson remains the author. Technical, human-factors, accessibility, privacy, safety, current-source, and founder review are mandatory before publication. Publication is not authorized.