Research Note
Video Tracking Task Selection Record
Choose a video method by defining the unit that must persist and the output consumed by the workflow. A demonstration should not determine the task.
Video Tracking Task Selection Record
Selection rule
Choose a video method by defining the unit that must persist and the output consumed by the workflow. A demonstration should not determine the task.
Task map
| Task | Persistent unit | Typical output | Important boundary |
|---|---|---|---|
| Optical flow | Local image motion between frames | Dense two-dimensional displacement field | Does not by itself preserve a long-term point identity |
| Point tracking | Arbitrary queried surface point | Trajectory and visibility | Does not identify the object or person |
| Multi-object tracking | Detected object instance | Box, class, track identifier | Depends on detection and can switch identities |
| Keypoint tracking | Defined semantic landmark | Landmark coordinates and confidence | Requires a known landmark schema |
| Video segmentation | Object or region | Pixel mask through time | Boundary and identity errors differ from point drift |
| Camera tracking | Camera relative to scene | Pose through time | Usually assumes or models scene geometry |
| 3D point tracking | Physical point in three-dimensional space | 3D trajectory | Requires depth, multi-view, or learned 3D reasoning |
Combined systems
A useful system may detect an object, segment it, sample points inside the mask, track those points, estimate camera motion, and feed the result into reconstruction or editing.
The correct evaluation unit follows the final decision. If the workflow needs a stable mask, a point metric alone is incomplete. If it needs a single surface correspondence, object-box accuracy may hide the relevant error.
Decision questions
The team should write one sentence naming what must persist, the required spatial precision, whether future frames are available, how occlusion is handled, the acceptable delay, and the cost of a wrong result.
Then it should identify the annotations, metrics, data rights, runtime constraints, and human review needed for that output.
Boundary
Task definitions describe common formulations. Research systems can blur them. The selection record is a design aid, not a claim that one method always dominates another.
Sources
Follow the evidence.
- youtu.be: BNTcjZ0Ym38youtu.be
- ai.meta.com: sam2ai.meta.com
- proceedings.neurips.cc: 58168e8a92994655d6da3939e7cc0918 Abstract Datasets and Benchmarksproceedings.neurips.cc
- arxiv.org: 2410arxiv.org
- open.spotify.com: 26JgnnwjvK5vYIdRofV8ntopen.spotify.com
- github.com: co trackergithub.com
- cotracker3.github.iocotracker3.github.io
- NIST AI Risk Management Frameworknist.gov
- vggsfm.github.iovggsfm.github.io
- daltonanderson.ghost.io: metas cotracker 3 a leap in ai object trackingdaltonanderson.ghost.io
- arxiv.org: 1803arxiv.org
- ecva.net: 3526 ECCV 2020 paperecva.net
- github.com: tapnetgithub.com
- raw.githubusercontent.com: LICENSEraw.githubusercontent.com
- tapvid.github.iotapvid.github.io
- NIST Privacy Frameworknist.gov
- arxiv.org: 1504arxiv.org
- openaccess.thecvf.com: Karaev CoTracker3 Simpler and Better Point Tracking by Pseudo Labelling Real Videos ICCV 2025 paperopenaccess.thecvf.com