TacBench Explained: How to Read the Sparsh Results

How to Read the TacBench Results

TacBench is a six-task benchmark for judging whether a pretrained tactile representation can support several kinds of downstream work. It covers force, slip, relative pose, grasp stability, textile recognition, and a bead-maze policy.

Its most repeated result is a 95.1 percent average improvement for Sparsh over task and sensor-specific end-to-end baselines. That number is not model accuracy. It combines relative comparisons across different tasks, sensors, metrics, and label budgets.

The right way to read TacBench is task first, metric second, baseline third, split fourth, and aggregate last.

flowchart TD
    A["Read one TacBench task"] --> B["Identify sensor and collected data"]
    B --> C["Identify label and held-out unit"]
    C --> D["Understand the metric and direction"]
    D --> E["Check frozen Sparsh decoder against matched E2E baseline"]
    E --> F["Inspect uncertainty and failure cases"]
    F --> G["Ask what new sensor, object, session, or robot remains untested"]
    G --> H["Only then interpret the aggregate"]

Why a multi-task benchmark matters

Tactile research is fragmented. One paper may estimate force from DIGIT images. Another may classify materials from GelSight clips. A third may use touch inside a manipulation policy. Different sensors and outputs make broad claims hard to compare.

The Sparsh paper introduced TacBench to test one representation family across several of these jobs. Most evaluations keep the pretrained encoder frozen and train a smaller task decoder with labels.

That frozen setup asks a focused question. Did self-supervised pretraining preserve information that a new supervised task can use efficiently?

It does not ask whether one frozen encoder can run a complete robot without calibration, labels, policy training, or control logic.

The six tasks do different jobs

The paper also includes T1A, a qualitative force-field visualization. It is related to force estimation but is not a seventh row in the six-task aggregate.

T1 tests force estimation

Force estimation predicts three axes of load from tactile images. The team collected data with hemispherical, sharp, and flat indenters and synchronized the sensor images with force measurements.

The appendix reports root mean squared error in millinewtons. It trains on sharp and spherical probes and evaluates on flat-indenter data. That is stronger than a random frame holdout because the indenter geometry changes.

The result still belongs to the reported force range, sensors, mounting, preprocessing, and reference instrument. It does not establish force accuracy after a gel replacement or under a new contact geometry.

T2 tests an imbalanced slip event

The slip dataset contains 125,000 samples, but only 13 percent are slip. The authors therefore report F1 instead of raw accuracy.

That choice is important. A classifier that predicts "no slip" for every sample would look strong on accuracy and fail at the actual job.

Slip labels are derived from a friction-cone model with an empirically estimated coefficient. The paper includes a failure case where that boundary appears inaccurate even though the tactile evidence suggests slip.

This means the label can fail as well as the model. A field test needs an independent way to review disputed events.

T3 turns pose regression into classification

The pose task estimates relative SE(2) change between an object and the sensor. The outputs are translation in two directions and rotation.

The authors discretize the values into bins and report multiclass accuracy. That metric can be useful for the chosen method, but it is not the same as continuous millimeter and degree error.

A product reader should ask how often the estimate lands in a neighboring bin, how that maps back to physical error, and whether the controller treats all wrong bins equally.

T4 and T5 reuse existing datasets

Grasp stability adapts the Feeling of Success dataset. The paper says the original dataset did not provide an official train-test split, so the authors created a randomized split using all objects.

That split supports the reported comparison. It does not prove generalization to unseen object families. An object-held-out split would answer a different question.

Textile recognition uses the Clothing Dataset with 20 classes and 4,467 short video clips. The paper reports that MAE performs particularly well, which may reflect the value of detailed pixel appearance for texture.

The official repository points readers to the original datasets and preprocessing scripts. Reproduction should preserve the exact split and data transformation.

T6 brings the representation into a policy

The bead maze uses 50 demonstrations and about 34,000 pairs of tactile images and robot joint angles. A Diffusion Policy receives Sparsh features plus proprioception and predicts changes in seven joint angles.

The paper reports about 16 percent lower trajectory error for DINO and I-JEPA features compared with end-to-end policy training. In real rollouts, it reports 20 to 53 percent more distance before failure, depending on the comparison.

It also states that none of the models completed the full maze. That is not a footnote to remove. It defines the gap between a relative improvement and task completion.

The project demonstration helps a reader understand the setup, but selected video does not replace the evaluation record.

The 95.1 percent aggregate needs a warning label

The abstract and Meta's announcement say Sparsh improves over end-to-end training by 95.1 percent on average under the reported limited-label settings.

Appendix D Table 13 lists six relative improvements and displays a 98.75 percent average. The accessible paper therefore contains an internal numerical discrepancy.

The public interpretation should not choose a favorite number. It should state that the authors headline 95.1 percent, note the appendix value, and direct the reader to the task-level results.

The aggregate is also dominated by very large relative changes for slip and pose under constrained labels. Grasp stability improves by a much smaller percentage. An arithmetic average gives each benchmark row one vote, regardless of physical consequence or metric scale.

No one should use the aggregate to predict improvement on a seventh task.

What TacBench does not establish

TacBench does not test every sensor unit, contact geometry, object population, lighting condition, gel age, contamination state, compute target, or robot.

It does not provide a safety certification. It does not prove that an estimate arrives within the control deadline. It does not resolve commercial rights for code, checkpoints, datasets, or hardware.

The benchmark is valuable because it turns a broad representation claim into several concrete tests. Its limits are part of that value.

A practical reading sequence

Start with the task that resembles your job. Reproduce its baseline and Sparsh configuration with the released artifact. Freeze the split and metric. Then add a held-out object, session, and sensor unit that reflects your actual claim.

If the task-level result survives, use [[How to Evaluate a Tactile Sensing Model]] to test drift, latency, physical consequences, and stop conditions. [[How Sparsh Learns Touch Representations]] explains why the pretrained encoder and supervised decoder must remain separate in the review.

The wider release story belongs with [[E044 Content Plan]] when E044 is published.

Editorial note

This benchmark explainer was developed with AI assistance from Venture Step E045 and the linked paper, code, dataset, project, and announcement sources. Dalton Anderson remains the author. Technical, source, reproduction, and founder review are mandatory before publication. Publication is not authorized.