How to Evaluate a Tactile Sensing Model

How to Evaluate a Tactile Sensing Model

Evaluate a tactile sensing model as part of a physical measurement system, not as an isolated score. The test should vary representative contacts, objects, sensor units, calibration states, environments, and timing conditions. It should connect each model error to the robot action that could follow.

The first deliverable is not a leaderboard. It is a written decision: go within a bounded operating envelope, revise and retest, or stop.

This guide supports research and controlled engineering evaluation. It is not a safety certification or permission to run closed-loop control around people.

flowchart TD
    A["Define physical task and unacceptable outcome"] --> B["Choose current baseline and reference instrument"]
    B --> C["Freeze grouped split, metrics, and thresholds"]
    C --> D["Collect representative and adversarial contacts"]
    D --> E["Measure model error, uncertainty, drift, and latency"]
    E --> F["Translate errors into robot consequences"]
    F --> G{"Decision"}
    G -->|Go| H["Shadow or isolated integration inside bounded envelope"]
    G -->|Revise| I["Change data, sensor, model, or protocol"]
    G -->|Stop| J["Record failed evidence and prevent reuse"]

Define the decision before the metric

Write one sentence that describes what the estimate will change. A slip estimate might increase gripper force. A pose estimate might trigger an in-hand correction. A force estimate might support offline inspection without controlling anything.

Then state the unacceptable outcome. The robot might crush a fragile object, drop a hazardous part, strike a person, or damage the sensor. The same numerical error can have different consequences across these cases.

Define the operating envelope in physical terms. Include object types, contact locations, load range, motion, speed, sensor configuration, environment, and update deadline.

The evaluation is out of scope outside that envelope.

Select a reference and a baseline

The ground-truth instrument must be more trustworthy than the model for the property being tested. Force may require a calibrated force-torque sensor. Pose may use motion capture or tracked markers. Slip may need an independent motion observation and a reviewed physical definition.

The Sparsh repository provides concrete examples. Its force data uses an ATI Nano17 reference. Its pose data synchronizes DIGIT frames with tagged object pose.

Those examples are starting points, not universal approval. Record the reference range, resolution, uncertainty, calibration, timing, coordinate frame, and known failure modes.

Choose a current baseline. It may be an analytic estimator, a simple threshold, a task-specific network, the sensor's native output, or a no-tactile controller. Give the candidate and baseline the same test contacts and decision logic where possible.

Freeze the protocol before comparing models

The test plan should name the artifact revision, preprocessing, input history, image size, decoder, checkpoint, compute target, batch behavior, and random seeds.

Freeze the data split and metric before looking at final results. Sequential tactile frames from one contact are highly correlated. A random frame split can place nearly identical moments in train and test.

Group by the unit that could leak information. That may be contact trajectory, object, session, sensor serial, gel surface, calibration cycle, operator, or maintenance state.

If the claim concerns new objects, hold out objects. If it concerns replacement devices, hold out complete sensor units. If it concerns long-term performance, hold out later sessions.

Build a variation matrix

Representative conditions estimate expected performance. Adversarial conditions find the edge of the operating envelope. Both are necessary.

The test should also include no-contact, partial-contact, overload, and ambiguous-contact cases. A system must know how it behaves when its input is invalid.

Match metrics to the physical output

Force estimation needs error by axis, load, contact location, object, and sensor unit. Report bias, RMSE or absolute error, tails, and calibration plots. A low average can hide a dangerous high-load slice.

Slip detection needs precision, recall, F1, event timing, false alarms, and missed-event duration. The Sparsh paper uses F1 because only 13 percent of its slip data is positive. Your class balance may differ.

Pose evaluation should include continuous translation and rotation error even when the model uses classification bins. Grasp stability needs results by object family and failure mode, not only aggregate accuracy.

For a policy, report task progress, completion, interventions, collision or overload events, recovery, and failure location. The bead-maze research reported better relative progress while also reporting that no model completed the full maze. That is the kind of boundary a decision record needs.

Measure the whole latency path

Backbone frames per second is not control latency.

Measure from the time a physical contact becomes observable through camera exposure, transport, preprocessing, encoder inference, decoder inference, queueing, controller consumption, and actuator response.

Report median and tail latency. Record dropped frames, stale estimates, clock drift, backpressure, memory pressure, warmup, and recovery after restart.

Test the failure response. The controller should not treat a late or missing estimate as fresh evidence.

Translate model errors into robot consequences

A technical reviewer can describe an error distribution. A robotics owner must explain what that distribution does to the machine.

Replay recorded model outputs through the decision logic before live control. Compare the intended command with a conservative reference. Identify every case where the candidate would increase force, move, or continue when the safer action would stop.

Then run in shadow mode or an isolated fixture. Keep independent stop controls, conservative mechanical limits, and human supervision appropriate to the risk.

Do not move directly from a public benchmark to an unconstrained live robot.

Record a real decision

A go decision names the allowed task, sensor units, calibration procedure, environment, thresholds, monitoring, fallback, and review date. It does not say the model is generally safe.

A revise decision identifies the failed slice and the next experiment. More data is not always the answer. The problem may be sensor geometry, reference quality, timestamping, preprocessing, or the control rule.

A stop decision records the unacceptable result and the artifact version. This prevents a later team from quietly reusing a failed model after the context has been forgotten.

The NIST AI Risk Management Framework can support broader governance language, but it does not replace robotics hazard analysis or task-specific safety engineering.

A compact test record

Use [[How to Build a Tactile Data Collection Plan]] before the first test run. If the model is Sparsh, [[How to Read the TacBench Results]] helps identify the closest source task and the claims that still require local evidence.

Readers comparing another perception model can pair this with [[How to Evaluate a Point Tracking Model]] from E046. The measurement discipline is similar even though the physical consequences differ.

Editorial note

This evaluation guide was developed with AI assistance from Venture Step E045 and the linked primary research, repository, dataset, and risk-management sources. Dalton Anderson remains the author. Robotics, safety, technical, source, and founder review are mandatory before publication. Publication is not authorized.