How to Build a Point-Tracking Test Set

How to Build a Point-Tracking Test Set

A useful point-tracking test set is organized around complete video sequences, physical surface trajectories, visibility, and the hard conditions the workflow will actually encounter.

Do not randomly split frames from the same source video. Adjacent frames share scene, subject, camera, and motion information, which creates leakage and an unrealistically easy test.

Lock the sequence split before comparing model failures. Once the team uses a hidden test clip to tune query selection, preprocessing, thresholds, or model choice, that clip belongs to development data.

flowchart TD
    A["Define target video population"] --> B["Review rights, consent, privacy, and retention"]
    B --> C["Group by source sequence and capture context"]
    C --> D["Lock train, development, and hidden test groups"]
    D --> E["Define query, trajectory, and visibility policy"]
    E --> F["Pilot annotation and measure disagreement"]
    F --> G["Annotate hard conditions deliberately"]
    G --> H["Quality review and adjudication"]
    H --> I["Versioned dataset card and frozen evaluation release"]

Define the target population

Describe the videos the product is expected to process. Include camera type, viewpoint, resolution, frame rate, compression, clip duration, lighting, scene, subject, motion, capture distance, and operating environment.

Avoid broad labels such as "real-world video." A fixed security camera, a handheld phone, a broadcast sports feed, a microscope, and a robot wrist camera produce different tracking problems.

Define the query population too. Users may click arbitrary visible surfaces. A system may sample a grid. A detector may choose object interiors. A keypoint model may provide semantic landmarks. The test queries should match the product.

The dataset should make important exclusions visible. If it contains no night scenes, long occlusions, water, reflections, or rapid camera motion, a passing score does not cover them.

Resolve rights and privacy before collection

Determine who owns each video, which use is permitted, whether model evaluation and annotation are covered, whether people gave required consent, and whether the data can be shared with annotators or vendors.

Record restrictions on screenshots, failure reels, derived trajectories, faces, locations, audio, metadata, and publication. A derived coordinate stream can still reveal behavior or location.

Public benchmark availability does not grant permission to use customer, employee, medical, surveillance, or web video. The TAP-Vid project notes that its Kinetics subset uses YouTube videos with licensing constraints that affect visualization and distribution.

Define retention, encryption, access, deletion, incident response, and reviewer locations before sensitive data enters the annotation system.

Use the sequence as the split unit

Group frames by source video and by any larger context that can leak information. The correct unit may be a capture session, subject, location, camera, event, production, or device.

If one subject appears in several clips recorded seconds apart, placing those clips in separate partitions may still leak appearance and motion. The grouping policy should match the generalization claim.

Create training, development, and hidden test groups. The hidden test should remain inaccessible to model selection and prompt or threshold tuning.

For a small dataset, use grouped cross-validation or repeated grouped splits rather than frame-level randomization. Preserve the group identifiers and split-generation seed.

Define what the point represents

An annotation must stay on one physical surface location. "Track the hand" is not precise enough because the hand contains many moving and rotating surfaces.

Define whether queries may land on boundaries, deformable surfaces, reflections, transparent objects, featureless regions, shadows, specular highlights, or image artifacts. State how to handle a point that has no stable physical correspondence.

For each query, record the query frame, coordinates, surface description, and reason for selection. The description helps adjudicators recognize when the label has slid to a nearby pattern.

Define visibility, occlusion, and out-of-frame states

Create an operational rule for visibility. Decide whether a partially visible surface counts as visible, how much evidence is required, and how blur affects the label.

Occlusion means another surface blocks the queried point. Out of frame means the point projects beyond the image boundary. A scene cut, severe blur, reflection, and self-occlusion may need explicit guidance.

Define re-entry. When a point returns after a long absence, the annotator must reconnect to the same physical location, not a similar pattern.

The TAP-Vid paper is a strong reference because it formalizes arbitrary surface tracking and includes visibility in the benchmark.

Use assisted annotation without trusting it blindly

Manual point trajectories are slow and difficult. Interpolation, optical flow, or a tracker can propose coordinates between human-reviewed keyframes.

The TAP-Vid team used flow estimates to compensate for easier short-term motion so annotators could focus on hard sections. The project reports faster annotation with flow assistance, while noting that the easy demonstration shown on its page is not representative of every clip.

Assistance can also anchor annotators to a wrong path. The interface should make it easy to reject the proposal, inspect frame by frame, label visibility, and resume after occlusion.

Store the proposed path separately from the accepted label if later error analysis needs to distinguish tool and human decisions.

Build a hard-case coverage matrix

Do not wait for rare conditions to appear by chance. Sample them deliberately.

Weight the matrix by product frequency and consequence. A rare condition with severe impact may deserve more test coverage than its natural frequency suggests.

Pilot the annotation policy

Give the same initial clips and queries to multiple annotators. Measure coordinate disagreement, visibility disagreement, correction time, and unresolved cases.

Review disagreements with a technical and domain owner. Update the written policy and add examples that show correct, incorrect, and genuinely ambiguous labels.

Repeat the pilot until annotators can apply the rule consistently. If expert reviewers cannot agree on the ground truth, the product may need an abstention state or a different output.

Run quality control at the trajectory level

Frame-level spot checks can miss a brief jump that corrupts the whole track. Render each trajectory through time.

Use automated checks for impossible coordinate ranges, missing frames, stale positions, large jumps, inconsistent visibility, and mismatched resolution. Treat them as review flags, not automatic proof of error.

Double-annotate a declared sample from every condition and annotator. Adjudicate disagreements without overwriting the original labels.

The DeepMind TAP repository provides benchmark metric code and datasets. Reuse the official implementation when the protocol matches instead of re-creating a metric from a prose description.

Freeze metrics and preprocessing

State the input resolution, coordinate convention, query mode, visibility threshold, and metric version. Preserve the code revision used to score the set.

If model input is resized or cropped, test the round-trip coordinate transform. A systematic scaling or offset error can look like model failure.

Include localization, visibility, and a joint metric. Add workflow measures such as long-occlusion recovery, drift rate, correction time, and downstream error where relevant.

Publish an internal data card

The Datasheets for Datasets paper provides a useful documentation model. A point-tracking card should state purpose, creators, collection, rights, consent, population, exclusions, splits, annotation policy, quality checks, metrics, privacy, access, retention, known gaps, and change history.

Record every label correction with the old value, new value, reason, reviewer, date, and affected evaluation release. Do not silently repair a benchmark after results are published internally.

Create a new version when videos, queries, labels, policy, metrics, or splits change. Preserve prior results so the team can explain why a model decision changed.

Keep the final test hidden

The final partition should be scored only after the team freezes the model, checkpoint, queries, preprocessing, and thresholds.

If the model fails, preserve the result. The team can return to development data, make changes, and later evaluate on a new untouched test release or through a governed limited retest.

The test set should remain capable of surprising the team. If every failure is familiar, the set may no longer be measuring generalization.

Use the completed set with [[How to Evaluate a Point Tracking Model]]. Carry its rights, known gaps, and split identity into [[How to Review Point Tracking for Deployment]].

Editorial note

This technical guide was developed with AI assistance from Venture Step E046 and the linked benchmark, annotation, repository, and dataset-documentation sources. Dalton Anderson remains the author. Technical, data-rights, privacy, domain, and founder review are required before publication or use. Publication is not authorized.