Point Tracking Explained: Trajectories, Visibility, and Occlusion

What Point Tracking Does in a Video

Point tracking estimates where a selected physical surface point appears across video frames and whether that point remains visible. The output is usually a two-dimensional trajectory with a visibility state for each frame.

It does not automatically identify the object, recognize a person, preserve an object-level identity, recover three-dimensional position, or predict where the object intends to go. Those results require additional models, geometry, data, or domain logic.

The simplest mental model is a dot placed on a moving sleeve. The tracker tries to keep that dot attached to the same patch of fabric as the arm moves, turns, passes behind the body, and becomes visible again.

flowchart LR
    A["Query frame and x,y location"] --> B["Same surface point in later frames"]
    B --> C["2D trajectory"]
    B --> D["Visible or occluded state"]
    B --> E["Model confidence or diagnostic"]
    C --> F["Editing, motion analysis, reconstruction, or control input"]
    D --> F
    E --> F
    F --> G["Larger workflow decides meaning"]

A query point starts the track

A point tracker receives a video and one or more queries. Each query identifies a frame and an image coordinate. In plain language, it says, "Start with this location in this frame."

The query may come from a human click, a regular grid, a detector, a segmentation mask, a keypoint model, or another system. The sampling method changes what the output represents.

A click on a shoe follows one chosen surface point. A grid over the entire frame can reveal broader motion. Points sampled inside an object mask can help describe deformation or relative motion, but the point tracker still does not own the mask or object identity.

The CoTracker3 paper defines the query with a frame index and two-dimensional coordinates. The model predicts a two-dimensional point for each frame, along with visibility and confidence.

A trajectory is a sequence of coordinates

For one query, the trajectory is the ordered set of estimated coordinates over time. Draw those coordinates on the video and the result looks like a moving dot or a trail.

The trajectory describes where the tracker believes the same surface point projects into each frame. It is not automatically a forecast. Offline trackers can use frames from both directions around the query. Online trackers process sequential windows and estimate forward as video arrives.

A future product may use the recent trajectory to predict motion, but that prediction is a separate task with its own uncertainty and evaluation.

Coordinates also require a declared reference. A track at the original pixel resolution, a resized model input, a crop, and a normalized coordinate system are different outputs. Production systems need one explicit schema and a tested transform back to the consumer's coordinate space.

Visibility tells the system whether the point can be seen

Visibility asks whether the physical surface point is observable in the current frame.

If a sleeve passes behind a table, the point still exists but is occluded. If it leaves the image boundary, it is out of frame. If the camera cuts to another scene, the original point is no longer observable in the same continuous sequence.

These states can look similar in a rendered output because the point is not visible. They create different review questions. Occlusion tests whether the tracker can use temporal and neighboring evidence. Out-of-frame motion tests boundary and re-entry behavior. A scene cut may require the system to terminate rather than invent continuity.

The TAP-Vid benchmark treats occlusion as part of tracking. Its real-video annotations and synthetic ground truth allow models to be evaluated on both position and visibility.

Re-entry exposes whether the track stayed coherent

When the surface becomes visible again, the tracker may reconnect to the correct location, attach to a similar-looking area, or remain stuck near the last visible coordinate.

Re-entry is important because a plausible dot can be wrong. On repeated patterns, uniforms, wheels, windows, water, or featureless surfaces, several locations may look equally likely.

The CoTracker3 project page shows both occlusion examples and failures on sky and water. That combination is useful. It demonstrates why a model can perform well on benchmark averages and still fail on the exact surface a workflow cares about.

Sparse and quasi-dense tracking answer different questions

Sparse tracking follows a limited set of deliberately selected points. It can be efficient and easy to inspect. The result depends heavily on where the queries begin.

Quasi-dense tracking samples many points, often with a regular grid. It provides a richer picture of scene motion and deformation. It also increases compute, memory, rendering, and review demands.

Dense optical flow is different. It typically estimates a displacement vector for each image location between frames or nearby times. A point tracker tries to preserve an arbitrary queried surface correspondence over a longer sequence, including visibility.

The outputs can support one another. Optical flow may help annotate or initialize tracks. Point trajectories may feed reconstruction or editing. The right choice depends on the persistent unit and downstream output.

Joint tracking lets points share evidence

Some trackers estimate each point independently. CoTracker models allow multiple point tracks to interact.

That interaction can help when one point becomes hidden while nearby or related points remain visible. It does not mean the model has been given a perfect object structure. The attention mechanism learns relationships among point trajectories from data.

The distinction matters because "joint" does not guarantee that every point on a car, person, or deforming surface stays grouped correctly. It is a model capability to test, not an object-level truth.

Online and offline modes use time differently

An online tracker processes video in windows as frames arrive. It is appropriate when the workflow cannot wait for the complete clip. The implementation may still use overlapping frames, buffering, and nontrivial latency.

An offline tracker can use a longer window and information on both sides of the query. It may recover difficult tracks more accurately, especially around occlusion. It also has higher memory demands and cannot support a decision that must be made before future frames exist.

The official CoTracker repository provides both modes and returns predicted tracks with predicted visibility. It describes the online implementation as more memory efficient and suitable for long streams, while strongly recommending GPU use locally.

How a point track becomes useful

A video editor can attach an effect to a surface track. A reconstruction system can use many correspondences across frames. A motion-analysis tool can derive displacement, velocity, or deformation features. A robot system can treat tracks as observations inside a larger control pipeline.

In each case, the point tracker is a component. The consumer decides which points matter, how to handle occlusion, whether to reject low-quality output, and what action follows.

The risk of misunderstanding the task grows with the consequence. A wrong visual effect may be annoying. A wrong track used in an autonomous or safety decision may cause harm. Benchmark accuracy alone does not validate the latter.

What to define before choosing a model

Write one sentence naming the persistent unit. State whether it is a surface point, every pixel's local motion, an object instance, a semantic landmark, a region mask, camera pose, or a three-dimensional point.

Then state the required precision, query source, visibility behavior, sequence length, causal delay, number of points, frame rate, resolution, hardware, and cost of an incorrect trajectory.

That definition prevents a beautiful point-tracking demo from becoming an answer to the wrong product question.

Continue with [[Point Tracking Optical Flow or Object Tracking]] to choose the visual primitive. Use [[How to Evaluate a Point Tracking Model]] when the required output is a point trajectory.

Editorial note

This explainer was developed with AI assistance from Venture Step E046 and the linked primary sources. Dalton Anderson remains the author. Technical, source, domain, accessibility, and founder review are required before publication. Publication is not authorized.