Venture Step E046: What CoTracker3 Actually Tracks

What E046 Learned From CoTracker3

Venture Step E046 explored a deceptively simple computer-vision problem: mark a point in one video frame and follow that same physical surface point through time, including when it becomes hidden.

CoTracker3 mattered because its authors reported a simpler model and training recipe that used pseudo-labeled real video to improve point tracking. The paper compared a model fine-tuned on 15,000 real videos with BootsTAPIR, which used 15 million real videos, and reported stronger results on the named benchmarks.

The episode's central instinct was right. Better data and a simpler training design can change what a model learns. Its terminology sometimes moved too quickly from points to objects, from occlusion to leaving the frame, and from research efficiency to production readiness. Those distinctions are the useful story now.

flowchart LR
    A["Select a surface point in one frame"] --> B["Estimate its coordinates over time"]
    B --> C["Estimate visibility and confidence"]
    C --> D["Handle motion, occlusion, and re-entry"]
    D --> E["Feed tracks into a larger workflow"]
    E --> F["Evaluate the complete workflow"]
    B -. "Not automatic" .-> G["Object identity, intent, or 3D meaning"]

The visual intuition that made the paper interesting

E046 began with a person walking through a video while a grid of colored points moved across the scene. That demonstration makes point tracking feel like object tracking because many points happen to land on the person's clothing and body.

The model is doing something narrower. A query contains a frame and image coordinate. The tracker estimates where the corresponding physical surface point appears in other frames and whether it remains visible.

That distinction matters. A point on a sleeve can be followed without the system knowing the person's name, classifying the whole body, or assigning one persistent object identity. A point can remain mathematically tracked while the downstream object interpretation is wrong.

The TAP-Vid benchmark project formalized this tracking-any-point task. It includes real videos with human point-track annotations and synthetic videos with automatically available ground truth. It also makes occlusion part of the problem rather than treating a hidden point as an ordinary localization error.

Occlusion is not the same as leaving the frame

The episode described occlusion as a point moving outside the camera view. Both cases make a point unobservable, but they are not the same.

Occlusion occurs when something blocks the camera's view of the surface point. A hand may pass behind a table, or a person's sleeve may pass behind the body. Out-of-frame motion occurs when the projected point leaves the image boundary.

Re-entry is the hard part. A tracker may need to reconnect the trajectory after the surface becomes visible again. The correct result depends on appearance, surrounding motion, temporal context, and how the model represents relationships among points.

The CoTracker3 project page highlights tracking through occlusion and also shows failure cases on featureless sky and water. Those examples are a useful counterweight to a polished demo. The phrase "track any point" describes the query formulation, not a guarantee that every point in every scene will be tracked correctly.

What pseudo-labeling changed

High-quality point trajectories are difficult to annotate by hand. The surface coordinate must remain consistent across many frames, including moments of blur, deformation, and occlusion. Synthetic scenes can provide precise ground truth, but they do not reproduce every property of real video.

The CoTracker3 paper used both. The model was first trained on synthetic Kubric data. Frozen teacher trackers then generated pseudo-labels for real, unlabeled videos. A student model learned from those trajectories.

This corrects one of the episode's stronger claims. CoTracker3 did not replace synthetic data with real video. Synthetic pretraining remained part of the recipe. Pseudo-labeled real video was used for fine-tuning.

The paper used multiple teacher models with different strengths. During a training batch, it randomly selected a frozen teacher rather than merging every teacher prediction. The paper argues that real-video diversity, a larger noisy training set, and complementary teacher behavior can help the student outperform individual teachers.

That does not mean the student inherits every strength and none of the weaknesses. The paper's stated limitation is teacher quality and diversity. Performance eventually saturates because the student cannot learn information the teachers do not provide.

What the 1,000-times-less-data claim means

The most memorable number in E046 was one thousand.

The paper compared CoTracker3 fine-tuning on 15,000 real videos with BootsTAPIR training on 15 million real videos. Under the reported benchmark setup, CoTracker3 used one thousand times fewer real fine-tuning videos and achieved stronger results.

That is a meaningful result. It is not a complete efficiency claim. The comparison does not by itself establish total training compute, energy, annotation cost, storage, inference cost, or production latency.

The paper also reported two times fewer parameters than the earlier CoTracker model and 27 percent faster operation than LocoTrack in the authors' evaluated setup. Those are research comparisons tied to a version, hardware, query count, resolution, and implementation.

The official repository is the better source for reproducibility. It provides checkpoints, demos, evaluation commands, and training scripts. It also says a GPU is strongly recommended for local use and documents distinct online and offline modes.

What "simpler" actually describes

The CoTracker3 authors simplified architecture components and the semi-supervised training recipe relative to recent trackers. They removed a global matching module, simplified correlation processing, and used a more direct teacher-student pipeline.

Simpler does not mean the system is simple. The model still extracts multi-scale features, computes correlations, iteratively updates trajectories, and uses a transformer to reason across time and queried points.

The episode's explanation of "4D correlation" also needs refinement. The paper's 4D correlation compares two local two-dimensional feature neighborhoods, one around the query point and one around the current track estimate. Joint reasoning across multiple point tracks happens through the model's attention structure.

Calling the fourth dimension "correlation between tracks" compresses two different ideas into one. The corrected version makes the model easier to understand, not less interesting.

Online and offline are workflow choices

CoTracker3 provides online and offline variants.

The online model moves through overlapping windows in the forward direction. It is more memory efficient and can handle long or unknown-length streams. The offline model uses the video as a single window, can reason in both temporal directions from the query, and reportedly handles occlusion better. Its frame count is limited by memory.

Online does not automatically mean real-time for a particular product. End-to-end latency depends on resolution, number of points, hardware, decoding, batching, queues, rendering, and the downstream consumer.

Offline does not simply mean slower. It means the model may use information that would not exist at the moment a causal decision must be made.

Later publication context

Dalton recorded E046 after the October 15, 2024 arXiv release. The paper later appeared in the ICCV 2025 proceedings, pages 6013 through 6022.

That later conference record strengthens the publication identity. It does not turn the benchmark into proof for an untested workflow.

The next question is evaluation

The episode ended with many possible uses, including editing, robotics, reconstruction, and motion analysis. Those are plausible areas where point tracks can become inputs. None is established by a tracker benchmark alone.

A product team should begin with the required output. Does the workflow need an arbitrary point trajectory, a dense motion field, an object box and identity, a body landmark, a mask, camera pose, or three-dimensional structure?

Then it should reproduce the relevant benchmark, build a held-out set of representative sequences, review failures as video, measure latency and memory, and test the downstream consequence.

That path leads from a compelling paper to evidence. Read [[What Point Tracking Does in a Video]] for the task definition, [[How CoTracker3 Works]] for the research design, and [[How to Evaluate a Point Tracking Model]] for the comparison workflow.

The related E044 episode introduced CoTracker3 during a wider Meta release review. When its public package is available, it should be linked here as the earlier demonstration and release context.

The E046 audio and video preserve the original December 2024 discussion.

Editorial note

This page was developed with AI assistance from the raw E046 transcript and the linked primary sources. Dalton Anderson remains the author. Technical, source, license, domain, and founder review are required before publication. Publication is not authorized.