Point Tracking Evaluation Framework

Point Tracking Evaluation Framework

Evaluation contract

An evaluation must state the tracker, checkpoint hash, code revision, model mode, query policy, frame resolution, preprocessing, window length, hardware, metric implementation, dataset version, and split.

Scores produced with different query modes, resolutions, support points, visibility rules, or multi-point grouping are not automatically comparable.

Evidence layers

TAP-Vid context

TAP-Vid includes real and synthetic videos, human and perfect synthetic annotations, and metrics for tracking arbitrary points. It is a strong common reference, not a substitute for workflow evidence.

Average Jaccard combines localization and occlusion behavior across thresholds. Position accuracy measures visible-point localization at declared thresholds. Occlusion accuracy measures visibility classification. The official implementation should define exact calculations.

Failure reel

Every aggregate score should link to sampled failure clips. The reel should cover fast and abrupt motion, long occlusion, re-entry, camera motion, blur, deformation, lighting change, repeated texture, featureless regions, reflective surfaces, crowded scenes, and long sequences.

Each clip needs query points, ground truth, prediction, visibility state, error category, severity, and workflow consequence.

Decision rule

The model should pass declared benchmark-reproduction tolerance, representative-set thresholds, runtime budgets, and high-severity error limits. A high mean score cannot compensate for an unacceptable failure in the region that drives the workflow.

Boundary

High-stakes domains require independent domain evidence and safety review. This framework does not validate a medical, autonomous, surveillance, labor, or other consequential use.