DEVOURED
SAM 3: Segment Anything with Concepts (GitHub Repo)
Meta Superintelligence Labs released SAM 3, a new unified foundation model for image and video segmentation that significantly improves open-vocabulary concept segmentation from text or visual prompts, outperforming SAM 2.
What: SAM 3, from Meta Superintelligence Labs (Nicolas Carion et al.), is a new image and video segmentation model that expands on SAM 2's capabilities by enabling exhaustive segmentation and tracking of open-vocabulary concepts from short text phrases or visual exemplars. It achieves 75-80% of human performance on the new SA-Co benchmark (270K unique concepts), driven by a new data engine that annotated over 4 million concepts and a new architecture with a presence token and decoupled detector–tracker design.
Why it matters: This advancement signifies a major step towards more generalized and versatile AI vision systems, moving beyond predefined categories to understand and segment any described concept, which is crucial for real-world applications requiring nuanced object recognition in images and video.
Takeaway: Developers working on computer vision applications, particularly those involving open-vocabulary object segmentation or tracking in images and videos, should investigate SAM 3 and its new capabilities for enhanced performance and concept coverage.
Deep dive
- SAM 3 is a new unified foundation model for promptable segmentation in images and videos, developed by Meta Superintelligence Labs.
- It significantly improves upon its predecessor, SAM 2, by introducing the ability to exhaustively segment all instances of open-vocabulary concepts specified by text or visual prompts.
- SAM 3 achieves 75-80% of human performance on the new SA-Co benchmark, which contains over 270,000 unique concepts.
- This breakthrough is powered by an innovative data engine that automatically annotated over 4 million unique concepts for training.
- The model features a new architecture with a "presence token" for better discrimination between similar text prompts and a decoupled detector–tracker design to minimize task interference.
- SAM 3.1, released on March 27, 2026, introduces "Object Multiplex" for faster joint multi-object tracking.
- Requires Python 3.12+ and PyTorch 2.7+ with CUDA 12.6+.
- Access to model checkpoints needs to be requested on the SAM 3 Hugging Face repo.
- The project provides examples for image and video segmentation, batched inference, and using SAM 3 as an agent.
- Two new image benchmarks (SA-Co/Gold, SA-Co/Silver) and one video benchmark (SA-Co/VEval) are released.
- The model has 848 million parameters and consists of a detector and a tracker sharing a vision encoder.
Decoder
- Foundation model: A large AI model trained on a vast quantity of data that can be adapted to a wide range of downstream tasks.
- Promptable segmentation: The ability of an image segmentation model to identify and segment objects in an image based on various prompts, such as text descriptions, points, bounding boxes, or masks.
- Open-vocabulary concept segmentation: The ability to segment objects described by any arbitrary text phrase, rather than being limited to a predefined set of categories.
- SA-Co benchmark: A new benchmark (Segment Anything with Concepts) introduced with SAM 3, containing 270,000 unique concepts for evaluating open-vocabulary segmentation performance.
- Presence token: A new architectural component in SAM 3 designed to improve the model's ability to distinguish between closely related text prompts.
- Decoupled detector–tracker design: An architectural approach where the object detection and object tracking components are designed to operate independently, reducing interference and improving scalability.
- DETR (Detection Transformer): A transformer-based object detection model.
- MLLM (Multi-modal Large Language Model): A large language model capable of processing and understanding multiple types of data, such as text and images.
Original article
Full article content is not available for inline reading.
Explore all 62 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi