Point Tracking Task Record

Point Tracking Task Record

Formal task

Given a video and a query that contains a frame index and two-dimensional location, a point tracker estimates the corresponding two-dimensional point location across frames. Modern tracking-any-point systems also estimate whether the queried surface point is visible.

The CoTracker3 paper additionally predicts confidence about whether a track is close to the unknown ground truth under the paper's training definition. Confidence is not ground truth and should be calibrated for the intended workflow.

Terms

What the output does not contain

A point trajectory does not automatically identify an object, classify a person, assign a persistent object identity, produce a segmentation mask, infer three-dimensional position, estimate camera pose, or predict intent.

Those results may be built from point tracks and other components. The pipeline and its errors must be evaluated as a whole.

Common hard cases

Point trackers can drift on repeated or weak texture, lose a trajectory during fast motion or blur, confuse reflections, mishandle deformation, misclassify visibility, or fail to recover after a long occlusion.

The CoTracker3 project page explicitly shows featureless sky and water as failure cases. That evidence is more useful than a claim that the model can track any visible pixel in every scene.

Metric boundary

TAP-Vid evaluates localization and occlusion with declared query modes and resolution conventions. A metric is interpretable only with the dataset, query sampling, threshold, visibility rule, preprocessing, model mode, and code version used to compute it.