How to Evaluate an On Device AI Claim

How to Evaluate an On Device AI Claim

Evaluate an on-device AI claim by testing one named task on one named device and software version while observing downloads, network traffic, permissions, latency, power, accuracy, and fallback behavior.

"On device" can accurately describe one stage while other stages still use cloud services.

flowchart LR
    A["Name task and device"] --> B["Record model and software"]
    B --> C["Test connected first use"]
    C --> D["Test offline and partial network"]
    D --> E["Observe data flow and fallback"]
    E --> F["Measure result and resource cost"]
    F --> G["Write bounded claim"]

Name the task

Do not test "AI on the phone."

Test transcription, image enhancement, object removal, summarization, wake-word detection, reply generation, or another observable task.

Break a feature into stages when necessary. Audio capture, speech recognition, reasoning, generation, text-to-speech, storage, and sync may occur in different places.

The claim should identify which stage is local.

Freeze the device state

Record device model, processor, memory, storage, battery state, operating system, security update, app version, account, region, language, feature flags, model files, and network.

First use may download a component that later works offline. A warm test alone can hide the dependency.

Account and regional checks may require a server even when inference is local.

Google's Pixel Help Center is a current support source. The Made by Google 2024 collection is a dated launch source. Neither replaces observation of the tested task.

Test connected first use

Begin from a documented clean or known state.

Record downloads, permission prompts, account checks, model installation, setup time, network endpoints where lawful to inspect, bytes transferred, and local storage changes.

Run representative inputs and preserve outputs and timing.

Do not infer that transferred bytes contain private content without packet or vendor evidence. Do not infer privacy merely because traffic volume is small.

Test offline

Disable network access in a controlled way and repeat the task.

Record whether the feature is available, degraded, delayed, queued, blocked, or silently changed. Restart the app or device if normal use can include a cold start.

Test multiple input sizes and supported languages.

An offline success strongly supports local execution for the tested path. It does not prove that no telemetry or sync occurs later when connectivity returns.

Test partial and poor connectivity

A feature may prefer cloud processing when connected and local processing when offline, or the reverse.

Throttle or interrupt connectivity where authorized. Observe timeouts, retries, duplicate requests, user messages, output differences, and recovery.

Record whether the interface tells the user that processing moved between device and cloud.

Fallback is part of the product behavior.

Observe data flow and permissions

Review requested permissions, privacy settings, account activity, logs available to the user, network traffic, local files, and vendor documentation.

Map which inputs, derived features, prompts, outputs, feedback, diagnostics, and identifiers leave the device.

Google Photos' current editing help illustrates why feature-level review matters. Some tools require downloads, minimum hardware, connectivity, personal accounts, opt-ins, Face Groups, or location-related settings.

That record does not establish the path for every edit. Test the exact feature.

Measure user-visible value and cost

Record task accuracy or quality, time to first result, total latency, battery change, heat, storage, memory pressure, failure rate, and accessibility.

Compare local and connected behavior if both paths exist. Use the same inputs and scoring method.

A local path can improve privacy and responsiveness while reducing quality or increasing power use. A cloud path can improve capability while increasing data exposure and dependency.

The evaluation should show the tradeoff rather than assume one architecture is always better.

Write a bounded conclusion

A defensible conclusion names the task, device, version, test date, network state, result, observed traffic, known setup dependency, and untested conditions.

"Transcription completed offline after a model download on this device and version" is stronger than "the phone's AI is private and on device."

Recheck after model, application, operating system, account, policy, hardware, or region changes.

Preserve the test inputs, outputs, logs, settings, and screen record where appropriate. A later reviewer should be able to distinguish observed behavior from a vendor statement and reproduce the conclusion without relying on memory.

Use [[Made by Google 2024 and Pixel 9 Product Record]] for the dated launch context. Use [[Google Photos AI Editing Product Record]] for current edit-specific requirements.

This evaluation guide was developed with AI assistance from E030, current Google support material, and the linked test framework. Dalton Anderson remains the author. Technical, privacy, performance, accessibility, current-source, and founder review are mandatory before publication or reliance. Publication is not authorized.