How to Evaluate a Point-Tracking Model

How to Evaluate a Point-Tracking Model

Evaluate a point tracker with two evidence sets: a reproducible public benchmark and a held-out set of videos that represents the real workflow.

The benchmark tells you whether the implementation behaves within an expected range under a shared protocol. The representative set tells you whether the tracker survives your motion, surfaces, cameras, occlusions, latency budget, and failure costs.

Do not choose a model from one average score. Build a failure reel that renders the predicted and reference trajectories so reviewers can see where the model drifts, disappears, or reconnects to the wrong surface.

flowchart TD
    A["Define workflow and costly failure"] --> B["Freeze artifact and evaluation contract"]
    B --> C["Reproduce a public benchmark slice"]
    C --> D["Run a held-out representative test set"]
    D --> E["Measure localization and visibility"]
    E --> F["Render and classify failure clips"]
    F --> G["Measure latency, memory, and throughput"]
    G --> H{"All declared thresholds pass?"}
    H -->|No| I["Reject, redesign, or narrow scope"]
    H -->|Yes| J["Proceed to bounded deployment review"]

Define the workflow before the metric

Write down what supplies the query point, which surface must persist, how long the trajectory matters, whether future frames are available, and what consumes the output.

Then name the expensive error. An editor may care most about visible jitter. A reconstruction system may care about long-term drift. A manipulation system may care about losing a point during occlusion. A human-review tool may tolerate an error if it is clearly flagged and easy to correct.

The workflow definition determines which clips, points, thresholds, and operating measurements belong in the evaluation.

It also prevents a benchmark from becoming the product objective. A model can score well on average and fail every transparent, reflective, low-texture, or long-occlusion sequence that matters to the product.

Freeze an evaluation contract

Record the model name, checkpoint hash, repository revision, dependencies, model mode, query policy, query grouping, support points, frame resolution, resize and crop behavior, coordinate transforms, window length, hardware, precision, batch size, and metric code revision.

This record is necessary because point-tracking numbers move when the protocol changes.

The official CoTracker repository notes that joint evaluation of all target points produces numbers similar to the paper, while exact reproduction may require a single-point flag. Joint and single-point execution can also have very different runtime cost.

Do not compare two published scores until the dataset split, query mode, input resolution, visibility rule, model mode, and metric implementation match.

Reproduce a public benchmark slice

TAP-Vid is the main shared reference for tracking any point. It includes real and synthetic video subsets, point trajectories, visibility annotations, and a semi-automatic annotation process.

The NeurIPS TAP-Vid paper defines the benchmark and evaluation design. The DeepMind TAP repository provides dataset and metric code.

Begin with a small official split that the model repository explicitly supports. Confirm that the result is within a declared tolerance of the source's reproducible range. If it is not, stop and investigate the implementation before interpreting your own data.

A mismatch may come from a wrong checkpoint, coordinate transform, query mode, resolution, visibility convention, model mode, dependency, or metric revision.

Understand the metrics

Point-tracking evaluation usually separates localization from visibility.

Position accuracy measures whether a predicted visible point falls within declared distance thresholds of the reference. Thresholds are often normalized to a benchmark resolution. A percentage is meaningless without those conventions.

Occlusion accuracy measures whether the model correctly classifies visible and occluded states. A high localization score on visible frames can coexist with poor visibility behavior.

Average Jaccard in TAP-Vid combines position and occlusion behavior across thresholds. It is useful because a model must place a point correctly and make the relevant visibility decision. The exact formula should come from the official implementation.

Add workflow metrics where needed. These may include drift per second, successful re-entry after occlusion, uninterrupted visible-track length, false reappearance, manual correction time, or downstream reconstruction error.

Build a representative test set

Sample from the actual camera, resolution, frame rate, compression, lighting, subjects, surfaces, scene density, motion, and sequence length the workflow will encounter.

Split at the sequence or capture-group level. Adjacent frames from the same video must not land in train, tuning, and test partitions.

Include hard conditions deliberately. Fast motion, abrupt changes, camera shake, zoom, motion blur, deformation, repeated texture, featureless surfaces, reflections, transparency, partial and long occlusion, out-of-frame motion, re-entry, crowded scenes, and long streams belong in the coverage matrix.

Keep the final partition hidden while selecting models and thresholds. If the team watches test failures and tunes the system against them, the set has become development data.

[[How to Build a Point Tracking Test Set]] provides the full annotation and lineage workflow.

Freeze the query policy

Query selection can change the problem.

A regular grid emphasizes broad surface coverage. Human-selected points may focus on easy or important areas. Detector-selected features may avoid ambiguous surfaces. Points sampled inside a segmentation mask condition the evaluation on another model.

Declare which frame supplies the query, whether queries begin only when visible, whether boundary points are included, how many points are sampled, and how points are grouped during inference.

If the product lets users click any visible pixel, an evaluation that samples only strong SIFT features is incomplete.

Render predictions as video

Aggregate metrics hide failure shape. Render the reference point, predicted point, trajectory history, visibility state, and query origin on the original video.

Create a failure reel organized by error type and consequence. Include localization drift, wrong visibility, loss during occlusion, wrong re-entry, background attachment, repeated-texture jump, stale coordinate, unstable jitter, and output after a scene cut.

Review clips at normal speed and frame by frame. A two-pixel oscillation may be harmless in a low-resolution benchmark and distracting in a visual effect. A brief wrong re-entry may corrupt a longer reconstruction.

Each clip should link to the model artifact, sequence, query, metric row, and reviewer decision.

Test long sequences and online behavior

A short benchmark clip may not expose accumulation, state resets, window boundaries, or queue growth.

For an online tracker, test streams longer than the training window. Examine behavior at overlapping-window boundaries, after dropped frames, during variable frame rates, and after pauses.

For an offline tracker, test memory growth with frame count and query count. Confirm whether the implementation resizes, truncates, or fails on sequences beyond its expected window.

The CoTracker3 paper says the offline variant is memory bound, while the online variant can process long streams through sliding windows. That is a model-level statement, not an end-to-end capacity guarantee.

Measure the full runtime path

Record decode, validation, resize, transfer, initialization, inference, coordinate conversion, rendering, storage, and downstream delivery.

Measure warm and cold latency, throughput, peak and steady memory, GPU utilization, CPU load, queue depth, dropped frames, and recovery after hardware or input failure.

Vary resolution, clip length, number of queries, query grouping, batch size, and concurrent streams. Report percentiles rather than one best-case average.

The model may be fast enough alone while the complete pipeline misses its deadline.

Convert errors into a decision

Create a comparison record for every candidate. Preserve the reason a model was rejected as well as the winning score.

A model passes only when every critical threshold passes. Do not average a severe safety or rights failure into a good mean score.

Preserve a rollback path

An evaluation decision should name the checkpoint, threshold, approved scope, owner, monitor, and stop condition. It should also identify the previous system or manual path that can replace the tracker.

A material change to the camera, resolution, point policy, model, checkpoint, preprocessing, hardware, population, or downstream decision requires renewed evaluation.

If the model passes, continue to [[How to Review Point Tracking for Deployment]]. Passing this guide is evidence for a bounded deployment review, not production authorization.

Editorial note

This evaluation guide was developed with AI assistance from Venture Step E046 and the linked benchmark, paper, and code sources. Dalton Anderson remains the author. Technical, data, domain, reproducibility, and founder review are required before publication or organizational use. Publication is not authorized.