How CoTracker3 Works: Pseudo-Labels, Point Tracks, and Modes

How CoTracker3 Works

CoTracker3 is a point-tracking model and semi-supervised training recipe. It starts with synthetic pretraining, uses frozen teacher trackers to generate point-trajectory pseudo-labels for real video, and fine-tunes a student model on those generated tracks.

At inference, the model receives video and query points. It computes multi-scale image features, compares local neighborhoods through 4D correlation, and iteratively updates each point's trajectory, visibility, and confidence with a transformer that can share information across tracks.

The research contribution is simpler relative to recent point trackers. It is not a claim that production deployment is simple.

flowchart TD
    A["Synthetic Kubric video with ground truth"] --> B["Pretrain teacher and student candidates"]
    C["Unlabeled real video"] --> D["Sample query points"]
    B --> E["Randomly select a frozen teacher per batch"]
    D --> E
    E --> F["Generate pseudo-labeled trajectories"]
    F --> G["Fine-tune CoTracker3 student"]
    G --> H["Online checkpoint"]
    G --> I["Offline checkpoint"]
    H --> J["Sequential sliding-window point tracks"]
    I --> K["Whole-window forward and backward point tracks"]

The label bottleneck

Point tracking needs precise trajectories. An annotator must keep the label on the same physical surface location across frames, including blur, camera motion, deformation, occlusion, and re-entry.

Synthetic video can provide exact labels because the renderer knows scene geometry and motion. That precision made synthetic data central to early deep point trackers. The drawback is a distribution gap. Real cameras, compression, lighting, texture, motion, and scenes do not perfectly match rendered training data.

The CoTracker3 paper addresses that gap without requiring humans to label every real trajectory. Existing trackers act as teachers and create noisy labels on real video.

This is semi-supervised learning through pseudo-labels. The labels are predictions, not verified truth. The training design succeeds only if teacher errors, point sampling, real-video diversity, and the student's learning behavior produce a useful signal.

Synthetic pretraining remains part of the method

CoTracker3 is sometimes summarized as a real-video alternative to synthetic data. The actual recipe uses both.

The teachers are trained on synthetic Kubric data. The student is also pretrained on synthetic data. Real, unlabeled videos are then passed through frozen teachers, and the resulting trajectories supervise fine-tuning.

The paper describes roughly 100,000 Internet-like videos in its collected pool, with varied scenes and dynamic subjects. The main comparison uses a 15,000-video subset for real-video fine-tuning.

The training mix matters because the phrase "trained on real video" can incorrectly imply that synthetic ground truth disappeared. It did not.

Multiple teachers provide different signals

The reported teacher set includes CoTracker3 online, CoTracker3 offline, the earlier CoTracker, and TAPIR. All were initially trained on synthetic data.

For each training batch, the pipeline randomly selects one frozen teacher. Across epochs, the same real video can receive trajectories from different teachers. The paper reports that randomly choosing a teacher worked better than averaging or taking the median of all teacher predictions.

The intuition is that teachers have complementary behavior. An offline tracker can use more temporal context and may handle occlusion better. An online tracker may stay close to the query near its origin. A student trained across those signals can learn a useful combination.

This is not a guarantee that teacher weaknesses disappear. The paper's explicit limitation is dependence on teacher quality and diversity. Scaling eventually saturates because the student has absorbed the available teacher knowledge.

Query points shape the training signal

A tracker needs both video and a point to follow. The pseudo-labeling pipeline samples query points with a preference for image features that are suitable for tracking.

The paper uses SIFT detections to find descriptive points in selected frames. If a video does not provide enough suitable points, the training pipeline skips it.

That choice improves label stability, but it also shapes the training population. Featureless surfaces are less likely to produce strong keypoints. The CoTracker3 project page shows sky and water as failure cases, which is consistent with the difficulty of tracking weak or ambiguous texture.

The paper's ablations report that several sampling methods worked similarly, with SIFT consistently strong across the evaluated TAP-Vid datasets. A product should still test its own surfaces rather than treating that result as universal.

Feature maps describe each frame

The model begins by using a convolutional neural network to compute feature maps for every frame at several scales.

These learned features are not object labels. They are numerical representations that help the model compare local image evidence across time. Multiple scales allow the network to examine fine and broader neighborhoods.

The input is downsampled for efficiency. That creates an important deployment detail: the model's internal feature resolution, input resize, and output-coordinate transform must be recorded if a downstream system expects precise original-frame coordinates.

4D correlation compares local neighborhoods

The phrase "4D correlation" can sound like a claim about space and time. In the paper, it refers to comparing two local two-dimensional feature neighborhoods.

One neighborhood is sampled around the query location in the query frame. The other is sampled around the current track estimate in another frame. Comparing every feature position in one two-dimensional neighborhood with every position in the other produces a four-dimensional correlation structure.

A multilayer perceptron compresses those correlations into features the transformer can use. This is distinct from cross-track attention, which lets multiple point tracks exchange information.

The distinction prevents an attractive but inaccurate explanation that the fourth dimension is simply "correlation between tracks."

Iterative updates refine the estimate

The model initializes the track at the query coordinates across frames, with visibility and confidence values that must be refined.

At each iteration, it uses local correlation evidence, temporal displacement encodings, visibility, confidence, and shared transformer context to predict updates. It moves the point estimates, updates visibility and confidence, resamples features around the new positions, and repeats.

This resembles solving a correspondence problem through repeated correction. It is not object reasoning in the human sense. The model has learned patterns that help refine trajectories.

Joint tracking lets points inform one another

CoTracker's name reflects joint point tracking. The transformer processes a grid spanning time and queried points, with attention across temporal and point dimensions.

Nearby or related trajectories can provide context when one point becomes hard to observe. That can improve occlusion behavior compared with fully independent tracking.

Joint processing also changes evaluation. The official repository notes that running target points jointly produces results similar to the paper, while exact reproduction of reported numbers may require a single-point setting. Query grouping, support points, and throughput can affect both score and cost.

An honest comparison records how queries were grouped.

Online and offline modes share architecture but not time access

The paper provides online and offline variants with the same broad architecture and different training and operating behavior.

The online model processes overlapping windows forward through time. It carries predictions from one window into the next. It is designed for sequential video and can continue through long or unknown-length streams.

The offline model treats the available video as one window and predicts both forward and backward relative to the query. The paper reports better handling of occluded and long-term visible points, while noting that the maximum frame count is memory bound.

The terms describe temporal access, not a universal speed ranking. Online still buffers frames and consumes compute. Offline may be faster for a bounded batch under some setup while remaining unusable for a causal decision.

What the reported numbers mean

The arXiv v1 paper reports that CoTracker3 fine-tuned on 15,000 real videos outperformed BootsTAPIR, which used 15 million real videos, on the named TAP-Vid comparisons. That is the basis of the one-thousand-times-less-data statement.

The same paper reports two times fewer parameters than the earlier CoTracker and 27 percent faster processing than LocoTrack in the evaluated setup.

These claims need their comparator and context. They do not establish one thousand times less total compute, two times lower memory in every mode, or 27 percent faster end-to-end product latency.

The paper later appeared in the ICCV 2025 proceedings. That conference record should anchor the publication identity, while benchmark numbers must come from the exact reviewed version and evaluation protocol.

Reproduction comes before adoption

The repository provides checkpoints, PyTorch Hub examples, evaluation commands, and training scripts. It strongly recommends GPU use. It also states that most of CoTracker is licensed under CC BY-NC, with portions under separate terms.

A team should first reproduce a narrow benchmark slice with the released checkpoint and code revision. It should then freeze query sampling, resolution, mode, and metrics before comparing its own videos.

Only after representative evaluation should the team measure end-to-end decoding, batching, memory, latency, queue behavior, failure detection, and downstream correction.

Use [[How to Evaluate a Point Tracking Model]] for that process. If the model passes, continue to [[How to Review Point Tracking for Deployment]] for the license, data, privacy, monitoring, and rollback gates.

Editorial note

This research explainer was developed with AI assistance from Venture Step E046 and the linked paper, conference, project, repository, and benchmark sources. Dalton Anderson remains the author. Technical, source, license, reproducibility, and founder review are mandatory before publication. Publication is not authorized.