SAM3

SAM 3 (Segment Anything with Concepts)

SAM 3 is a unified model that detects, segments and tracks all instances of a concept in images and videos.

• Paper: Carion, Gustafson, Hu, Debnath, Hu, Suris, Ryali, Alwala, Khedr, … Dollár, Ravi, Saenko, Zhang, Feichtenhofer (Meta Superintelligence Labs), SAM 3: Segment Anything with Concepts, November 2025, code, website.
• Core idea: where SAM and SAM2 segment one object per prompt from a point/box/mask, SAM 3 introduces Promptable Concept Segmentation (PCS): a prompt is a short noun phrase (“yellow school bus”), one or more image exemplars, or both, and the output is a mask plus a persistent identity for every matching instance.
• Architecture: a DETR-style detector paired with a SAM 2-style tracker over a shared Perception Encoder backbone, with a presence head that decouples recognition from localization.
• Data: trained on data from a human- and AI-in-the-loop data engine producing 4M unique concept labels.
• Headline result: roughly doubles the accuracy of the best existing systems on the new SA-Co benchmark while also improving on SAM 2’s visual segmentation tasks.

SAM 3 keeps SAM 2’s promptable visual segmentation with clicks (left) and adds promptable concept segmentation (right): noun phrase and/or positive/negative image exemplars, applied to every instance.

The PCS task

Given an image or short video ($\le 30$ s), detect, segment and track all instances of a visual concept specified by a short text phrase, image exemplars, or a combination. Concepts are restricted to simple noun phrases — a noun plus optional modifiers — not referring expressions or reasoning queries. Noun-phrase prompts are global to all frames of the media; image exemplars are given on individual frames as positive or negative bounding boxes, and are used to refine the target masks iteratively. Individual mask(let)s can additionally be refined with PVS-style clicks in the manner of SAM2.

The task is intrinsically ambiguous: polysemy (“mouse”), subjective descriptors (“cozy”, “large”), ungroundable phrases (“brand identity”), boundary ambiguity (does “mirror” include the frame?), occlusion and blur. SAM 3 attacks this from three sides — an evaluation protocol that allows multiple valid interpretations (three independent annotations per datapoint on SA-Co/Gold), annotation guidelines that minimize ambiguity, and an ambiguity head in the model.

Compared with the SAM lineage: PVS (SAM 2) is “one prompt → one object, propagated”; PCS is “one concept → all objects, with identities”. SAM 3 supports both.

Architecture

Detector and tracker share the Perception Encoder image encoder. Cyan = from PE, blue = from SAM 2, yellow = new in SAM 3.

Detailed view: the detector’s dual encoder–decoder (fusion encoder, multimodal decoder, exemplar encoder, presence token) on the right; the video path with masklet matcher, memory attention and per-object memory bank on the left.

Backbone

Vision and text encoders come from the Perception Encoder (PE), contrastively trained on 5.4B image–text pairs, giving aligned image and text representations — SAM 2’s MAE-pretrained Hiera has strong localization but no linguistic grounding. Ablation (Table 11, SA-Co/Gold cgF${}_1$): PE-L+ 43.2, PE-L+ with an unaligned DistilRoBERTa text encoder 38.1, DINOv2-L 35.3, Hiera-L 32.8; PE also gives the best robustness under domain shift (COCO-O AP 42.5 vs 22.0 for Hiera-L).

Vision encoder: L+ size (450M params; text encoder 300M), windowed attention with global attention in only 4 of 32 layers, RoPE plus windowed absolute positional embeddings, input 1008 px split into $3\times 3$ non-overlapping 336 px windows. Tokens are downsampled $2\times 2$ to 1296 tokens before the fusion encoder. The text encoder is causal with a maximum context length of 32. As in SAM 2 the backbone runs once per frame for the whole interaction, streaming.

Detector

A DETR-family (DETR) dual encoder–decoder:

• Geometry and exemplar encoder. Each image exemplar (box + positive/negative label) is encoded from a positional embedding, a label embedding and ROI-pooled visual features, concatenated and processed by a small transformer. PVS visual prompts (points, boxes) are encoded the same way as “geometry tokens”. Geometry/exemplar tokens self-attend and cross-attend to the unconditioned frame embeddings.
• Fusion encoder. 6 transformer blocks with self- and cross-attention conditioning the unconditioned frame embeddings on the prompt tokens (text tokens + exemplar tokens), followed by an MLP. Vanilla self-attention. Output: conditioned frame embeddings.
• Decoder. 6 transformer blocks; $Q=200$ learned object queries self-attend, cross-attend to the prompt tokens and to the conditioned frame embeddings, then an MLP. Cross-attention uses box-region positional bias. Standard strong-DETR machinery is used: iterative box refinement, look-forward-twice, hybrid matching and DAC-DETR dual supervision, plus the Align loss. Each layer predicts a binary classification logit (does this query match the prompt?) and a box delta. 4 extra geometric queries handle the PVS task, playing the role of SAM 1/2’s output tokens.
• Segmentation head. Adapted from MaskFormer; semantic and instance segmentation share it. Conditioned fusion-encoder features produce the semantic mask; instance masks additionally use the decoder’s object queries. Multi-scale features come from a SimpleFPN since the encoder is single-scale.

Presence head

Forcing each object query to both recognize (what) and localize (where) is counterproductive: recognition needs global context, localization is inherently local. SAM 3 factorizes the prediction with a single learned global presence token, shared by all queries:

$p({\text{query}}_i\text{ matches NP})=p({\text{query}}_i\text{ matches NP}\mid \text{NP appears in image})\cdot p(\text{NP appears in image})$

The presence token is fed through an MLP classification head to give $p(\text{NP appears in image})$; each query only solves the conditional localization problem. At inference the total object score is the product of the presence score and the object score. Supervision strategy matters (Table 10): object/mask scores are supervised only when the concept is present in the image (unlike typical DETR variants, which supervise all object scores negatively for absent concepts), and presence and object scores are supervised separately — jointly calibrating them is slightly better on cgF${}_1$ but less flexible (it prevents conditioning on a concept known to be present, e.g. in counting, where one can simply set $p(\text{NP present})=1$). The presence token is concatenated to the object queries everywhere but is excluded from DAC. Presence training is a post-training step (Stage 3). Ablation: +1.5 cgF${}_1$ and +0.05 IL_MCC on SA-Co/Gold.

Ambiguity head

Without explicit handling, a phrase with two valid interpretations makes the loss-minimizing solution “output both at 50% confidence” — conflicting, overlapping masks. SAM 3 adds a mixture of $K$ experts trained winner-takes-all, as in SAM 1/2’s multi-mask output. For experts producing outputs $y_k$ with losses ${\mathcal{L}}_k$:

${\mathcal{L}}_{\text{MoE}}={\sum }_{k=1}^K{p}_k{\mathcal{L}}_k,\frac{\partial {\mathcal{L}}_{\text{MoE}}}{\partial {\theta }_j}=p_j\frac{\partial {\mathcal{L}}_j}{\partial {\theta }_j}$

$k^{\star }=\arg {\min }_k{\mathcal{L}}_k,{\mathcal{L}}_{\text{WTA}}={\mathcal{L}}_{k^{\star }},\frac{\partial {\mathcal{L}}_{\text{WTA}}}{\partial {\theta }_j}=\{\begin{array}{ll}\frac{\partial {\mathcal{L}}_{k^{\star }}}{\partial {\theta }_j},& j=k^{\star }\\ 0,& \text{otherwise}\end{array}$

$K=2$ works best; $K>3$ collapses. A separate classification head, trained with cross-entropy on top of a frozen SAM 3 to predict which expert took the minimal loss, selects the expert at inference — it adjusts only classification logits, leaving masks, boxes and presence scores unchanged. Overlapping instances are detected with Intersection-over-Minimum (better than IoU for nested instances); the ambiguity head gives a 15% reduction in overlapping instances.

Tracker and video

The tracker is SAM 2’s: prompt encoder, mask decoder, memory encoder and a per-object memory bank, with memory attention conditioning the current frame on spatial memories and object pointers. It shares the PE backbone (through a separate neck) with the detector, and is trained with the backbone frozen. An occlusion head predicts visibility, and only frames where the object is confidently present are retained in memory. Three masks per tracked object per frame are predicted with confidences; the most confident is used.

Per frame the detector and tracker run in parallel and are merged:

${\hat{\mathcal{M}}}_t=\text{propagate}({\mathcal{M}}_{t-1}),{\mathcal{O}}_t=\text{detect}(I_t,P),{\mathcal{M}}_t=\text{match\_and\_update}({\hat{\mathcal{M}}}_t,{\mathcal{O}}_t)$

Matching is IoU-based; unmatched detections spawn new masklets. The decoupling is deliberate: the detector must be identity-agnostic, the tracker’s job is to separate identities — the classic detection/tracking task conflict.

Temporal disambiguation. Define a frame-wise match indicator for masklet $i$ on frame $\tau$

${\Delta }_i(\tau )=\{\begin{array}{ll}+1,& \exists d\in {\mathcal{D}}_{\tau }\text{ s.t. }\mathrm{IoU}(d,{\hat{\mathcal{M}}}_{\tau }^i)>\text{iou\_threshold}\\ -1,& \text{otherwise}\end{array}$

and the Masklet Detection Score $S_i(t,t^{\prime })={\sum }_{\tau =t}^{t^{\prime }}{\Delta }_i(\tau )$. Four mechanisms build on it:

• Track confirmation delay. Output at frame $\tau$ is shown only after observing frame $\tau +T$ ($T=15$, ~half a second at 30 FPS).
• Removal of unconfirmed masklets. A new masklet with $S_i(t,t+T)<V$ (default $V=0$, i.e. matched on at least half the frames of the window) is dropped.
• Removal of duplicates. Two masklets both matching the same detection for $\ge \lceil T/2\rceil$ frames: the later-started one is removed. Empirically $\text{iou\_threshold}=0.1$ works best.
• Masklet suppression. A confirmed masklet whose lifetime MDS $S_i(t_{\text{first}}^i,\tau )$ drops below zero has its output zeroed but is retained in the tracker state, so it can be re-confirmed later.

Plus two re-prompting strategies. Periodic re-prompting: every $N=16$ frames, a detection with $\mathrm{IoU}(d,{\hat{\mathcal{M}}}_{\tau })\ge 0.8$ and both detection and masklet scores above 0.8 re-initializes the tracker for that object from the detection mask. Detection-guided re-prompting: on every frame, if the best-matching detection has box IoU $<0.85$ with the tracker’s prediction, the tracker is reconditioned on the detector output — countering mask drift/leak.

Training

Four stages (Table 18 lists the data per stage):

1. PE pre-training — image and text encoders on 5.4B image–text pairs; no distillation, no video fine-tuning.
2. Detector pre-training — the image-level detector plus vision/text encoders on large-scale pseudo-labelled and human-annotated (image, NP, masks) data, including video frames as images. Also pre-trains the visually prompted PVS task via the 4 geometric queries. To prepare for PCS, (image, NP) pairs are randomly converted to visual queries ($p=0.2$, NP dropped) or augmented with input boxes ($p=0.2$). PVS interactivity is limited to 4 steps for efficiency; the Align loss replaces SAM 2’s IoU prediction loss.
3. Detector fine-tuning — only the highest-quality exhaustivity-verified data (SA-Co/SYN dropped). Interactivity is expanded: 7 steps for PVS, and PCS gains interactive positive/negative exemplars sampled from model error (positive boxes from false negatives, negative boxes from high-confidence false positives with little ground-truth overlap), accumulated over 5 iterations. This stage introduces the presence token and presence loss.
4. Tracker training — decoder trained on top of the frozen backbone, on video, followed by a longer-temporal-context fine-tune.

Losses: boxes use ${\ell }_1$ + gIoU (weights 5 and 2), classification weight 100, masks focal + dice (weights 200 and 10), presence binary cross-entropy (weight 20). The tracker uses focal + dice mask loss, MAE for IoU prediction and cross-entropy for occlusion at 20 : 1 : 1 : 1, supervising only the lowest-loss mask among multi-mask predictions, and always supervising the occlusion head even when a frame has no ground-truth mask.

Augmentation. Geometric: crop/resize to a fixed $1008\times 1008$ square (aspect ratio not preserved), padding randomly distributed over all sides, random horizontal flip where left/right is meaningless. Semantic: a Wikidata mapping is used to sample synonyms (vocabulary expansion), sample negatives from ontology nodes absent from the image (only on exhaustively annotated data), and enforce hierarchy closure (if “canoe” and “boat” are both annotated, every canoe must also be labelled “boat”). Safety: slurs and subjective/non-visual adjectives (“a smart person”, “a dull person”) are sampled as negatives at train time. Mosaics up to $3\times 3$, only on datasets with low chance of missing annotations, merged via the Wikidata mapping.

Hyperparameters

Transcribed from §C.2 and §C.4 (no single hyperparameter table exists in the paper).

Model

Input resolution	$1008\times 1008$
Vision encoder	PE L+, 450M params, windowed attention, global attention in 4/32 layers, $3\times 3$ windows of 336 px, RoPE
Text encoder	PE, 300M params, causal, max context length 32
Tokens into fusion encoder	1296 (backbone tokens downsampled $2\times 2$)
Fusion encoder	6 transformer blocks, self + cross-attention, vanilla attention
Detector decoder	6 transformer blocks, box-region positional bias
Object queries	$Q=200$ (+ 4 geometric queries + 1 presence token)
Ambiguity experts	$K=2$ (winner-takes-all)
Dropout (encoder/decoder)	0.1

Stage 2 — detector pre-training

Iterations	~95k
Batch size	896
Optimizer	AdamW
Warmup / cooldown	5k / 5k steps
LR schedule	reciprocal square-root
Initial LR	5e-4 (vision), 1e-4 (text encoder), 1e-3 (all other components)
Layer-wise LR decay	0.9 (vision encoder)
Weight decay	0.1
Box losses	${\ell }_1$ (5), gIoU (2)
Classification loss	weight 100
Mask losses	focal (200), dice (10)
PVS interactivity	4 steps
PCS augmentation	$p=0.2$ convert to visual query, $p=0.2$ add input boxes

Stage 3 — detector fine-tuning

Iterations	~5k
Batch size	512
Learning rates	Stage-2 values $\times 0.025$
Presence loss	binary cross-entropy, weight 20
Interactivity	7 PVS steps; 5 iterations of exemplar sampling for PCS
Data	highest-quality exhaustivity-verified only (SA-Co/SYN dropped)

Stage 4 — tracker training

Iterations	~190k (+ 60k long-context fine-tune)
Batch size	512
LR schedule	cosine, peak 5.0e-4, linear warmup 1k iterations
Long-context fine-tune	16- and 32-frame videos, LR scaled by 0.1
Losses (weights)	focal+dice mask : MAE IoU : CE occlusion = 20 : 1 : 1 : 1
Backbone	frozen

Inference defaults: track confirmation delay $T=15$ frames, unconfirmed-masklet threshold $V=0$, matching $\text{iou\_threshold}=0.1$, periodic re-prompting every $N=16$ frames at confidence 0.8 and IoU 0.8, detection-guided re-prompting when box IoU $<0.85$. On an H200, a single image with 100+ detected objects takes 30 ms; video latency scales linearly with the number of tracked objects, giving near-real-time for ~5 concurrent objects, or 30 FPS for up to 10 objects on 2 H200s, 28 on 4, and 64 on 8.

Data engine

Media and ontology → mine data → propose NPs → propose masks with SAM 3 → AI + human verification → correction. Failed candidates go to humans; passing ones go straight into the dataset; the data retrains SAM 3 and the verifiers.

The engine mines (media, phrase) pairs on which the current SAM 3 fails. An AI model proposes noun phrases, another (SAM 3 itself, after Phase 1) proposes candidate instance masks, and a two-step verification follows: Mask Verification (MV) — accept/reject each mask on quality and relevance to the NP; Exhaustivity Verification (EV) — are all instances of the NP masked? Pairs failing EV go to human correction (adding/removing/editing masks with SAM 1 in a browser tool, or “group” masks for small inseparable objects). Ungroundable or ambiguous phrases can be rejected outright.

The key innovation over SAM 1/2’s engines is the AI annotator. Llama 3.2 is fine-tuned on the human accept/reject labels from Phase 1 (pre-trained on >200M image–text pairs from both tasks merged, then ~10M high-quality per-task points) into MV and EV verifiers that answer a multiple-choice question over a rendered image–phrase–mask triplet. They match or beat human accuracy (Table 23, average: MV 77.7 vs 75.0 human; EV 81.1 vs 81.8 human) and roughly double engine throughput. Table 15: total time per datamix item falls 90 s (human from NP) → 23 s (full engine); negative phrases 71 s → 6 s (5×), positive phrases 236 s → 152 s (36%).

Phase	Data	Domains	SA-Co/Gold cgF${}_1$	SA-Co/Silver cgF${}_1$
1 Human verification	1.2M images, 4.3M image-NP (SA-Co/HQ)	1	–	–
2 Human + AI verification	+122.2M image-NP	5	21.4	18.9
3 Scaling & domain expansion	+19.5M image-NP HQ; 39.4M images / 1.7B image-NP SYN; 9.3M / 136.6M EXT	15	54.4	50.5
4 Video annotation	52.5K videos, 134.3K video-NP	–	54.5	50.1 (VEval pHOTA 63.9)

• Phase 1. MetaCLIP images, BLIP-2 captions parsed with spaCy into NPs; masks from an off-the-shelf open-vocabulary detector (FIBER/OWLv2) prompting SAM 2; all verification human. SAM 3 is trained on this and becomes the next phase’s mask proposer.
• Phase 2. Ontology-guided retrieval mining (offline concept indexing from Wikimedia reference images + online image/text KNN retrieval over MetaCLIP), image-type balancing via an MLLM classifier, Llama 3.2 captioner + Llama 3.1 parser for NPs, removal of non-groundable phrases, NP balancing and cleaning, and a hard-negative proposer: negatives are proposed either from ontology siblings/cousins/uncles (“gray Siamese cat” → “tabby cat”, “dog”, “Chihuahua”) or by Llama 4, then checked for adversariality — a candidate is kept only if SAM 3 actually predicts masks for it that overlap the positive annotations. SAM 3 was retrained 6 times in this phase.
• Phase 3. Domain expansion to 15 datasets, mining crowded scenes and very small objects from SA-1B via a “crowdedness” metric, long-tail concepts from alt-text and from the 22.4M-node SA-Co ontology (Wikidata-derived, 17 top-level categories, 72 sub-categories). SAM 3 retrained 7 times, AI verifiers 3 times.
• Phase 4 (video). Aggressive mining for hard clips: scene-boundary detection + VMAF motion scores + PySceneDetect for non-static single-shot 5–30 s clips, video-ontology content balancing towards crowded/grouped scenes, an MLLM (PLM) judge doing video-QA to select challenging tracking scenarios, and PE-embedding similarity search seeded by known-hard videos. Frame-level captioning proposes NPs; masklets come from SAM 3, deduplicated by IoU, with whole-scene masklets filtered out; NPs with fewer than 3 objects and videos with fewer than 2 such NPs are discarded. Humans verify groundability and trackability, correct masklets with SAM 2 in the loop, and run a final exhaustivity confirmation.

SA-Co dataset and benchmark

Example video (top) and images (bottom) from SA-Co with annotated phrases and instance masks/IDs.

Training data — three image datasets and one video dataset:

Subset	Images/Videos	Unique NPs	Media-NP pairs	Masks/Masklets	% negatives
SA-Co/HQ (data engine, human+AI verified)	5.2M	4.0M	146.1M	52.3M	88.5%
SA-Co/SYN (Phase-3 engine, no humans)	39.4M	38.0M	1.7B	1.4B	–
SA-Co/EXT (existing datasets, ontology-enriched with hard negatives)	9.3M	497.4K	136.6M	70.5M	71.8%
SA-Co/VIDEO	52.5K videos	24.8K	134.3K	467.1K masklets	26.7%

SA-Co/HQ’s 4M unique NPs make it the largest high-quality open-vocabulary segmentation dataset — for comparison, LVIS has 1.2K NPs and Open Images 600.

Benchmark: 207K unique phrases, 121K images and videos, >3M media-phrase pairs, with hard negative labels to test open-vocabulary recognition. Splits: SA-Co/Gold (7 domains — MetaCLIP, SA-1B, Attributes, Crowded Scenes, Wiki-Common, Wiki-Food&Drink, Wiki-Sports Equipment — 51.8K NPs, 15.8K images, 168.9K image-NP, 84.4% negatives, three independent annotators for measuring human performance), SA-Co/Silver (10 domains, one annotation), SA-Co/Bronze and SA-Co/Bio (existing datasets), and SA-Co/VEval (video, 3 domains: SA-V, YT-Temporal-1B, SmartGlasses; 5.2K NPs, 1.7K videos, 75.4% negatives). SA-Co/Gold has ~$50\times$ more unique phrases than LVIS-test.

Metrics

AP is a poor fit: over huge label spaces most labels appear a handful of times, full PR curves are infeasible, and AP ignores calibration. SAM 3 therefore evaluates only predictions with confidence above 0.5 and splits the task into localization and classification.

Localization — on positive datapoints only, bipartite-match predictions to ground truth by IoU at threshold $\tau$, count TP/FP/FN, accumulate globally and compute positive micro F1:

${\text{pmF}}_1^{\tau }=\frac{2{\text{TP}}_{total}^{\tau }}{2{\text{TP}}_{total}^{\tau }+{\text{FP}}_{total}^{\tau }+{\text{FN}}_{total}^{\tau }},pm{F}_1=\frac{1}{10}{\sum }_{\tau \in \text{linspace}(0.5,0.95,10)}{\text{pmF}}_1^{\tau }$

Classification — a purely image-level binary decision (“is the object present?”), ignoring mask quality: a positive datapoint with any predicted mask above confidence 0.5 is IL_TP, otherwise IL_FN; a negative datapoint with any mask is IL_FP, otherwise IL_TN. Because negatives dominate, the imbalance-robust Matthews Correlation Coefficient is used:

$\text{IL\_MCC}=\frac{\text{IL\_TP}\cdot \text{IL\_TN}-\text{IL\_FP}\cdot \text{IL\_FN}}{\sqrt{(\text{IL\_TP}+\text{IL\_FP})(\text{IL\_TP}+\text{IL\_FN})(\text{IL\_TN}+\text{IL\_FP})(\text{IL\_TN}+\text{IL\_FN})}}$

The headline metric is the classification-gated F1:

${\text{cgF}}_1=100\cdot {\text{pmF}}_1\cdot \text{IL\_MCC}$

On SA-Co/Gold, predictions are scored against all three annotations and the best is taken (oracle protocol). Human performance itself differs sharply between protocols (Table 29: oracle 76.2 vs random-pair 55.5 cgF${}_1$), quantifying how ambiguous PCS is.

Results

Image PCS with text (Table 1). SAM 3 vs. the best baseline: LVIS zero-shot mask AP 48.5 (DINO-X 38.5), LVIS cgF${}_1$ 37.2 (LLMDet-L 35.1), SA-Co/Gold cgF${}_1$ 54.1 vs OWLv2${}^{\star }$ 24.6 — more than double, and 74% of the estimated human 72.8. Box detection: COCO AP 56.4, COCO-O 55.7, LVIS AP 53.6. Open-vocabulary semantic segmentation beats the strong specialist APE-D: ADE-847 13.8 mIoU (9.2), PC-59 60.8 (58.5), Cityscapes 65.2 (44.2).

With exemplars (Table 3). One exemplar box: AP${}^{+}$ on COCO 76.8 (image-only) / 78.1 (text+image), LVIS 76.0 / 78.4, ODinW13 82.2 / 81.8 — beating T-Rex2 by +18.3 / +10.3 / +20.5.

cgF${}_1$ vs. number of interactive box prompts, averaged over SA-Co/Gold phrases: concept-level exemplars generalize, so PCS climbs faster than an idealized per-instance PVS baseline.

Interactive PCS (Fig. 7). After 3 exemplars, interactive PCS beats text-only by +21.6 cgF${}_1$ and beats an ideal PVS correction baseline by +2.0 — exemplars generalize (they suppress or detect similar objects) whereas PVS fixes one instance at a time. Performance plateaus after 4 clicks, since exemplars cannot repair poor-quality masks; switching to PVS at that point yields further gains, showing the two modes are complementary.

Few-shot and counting. ODinW13 AP${}_0$ 61.0 / AP${}_{10}$ 71.8 and RF100-VL 15.2 / 36.5 — state-of-the-art 10-shot, beating in-context prompting in Gemini and gDino, with no prompt tuning. CountBench MAE 0.12 / accuracy 93.8% and PixMo-Count 0.21 / 86.2%, beating Gemini 2.5 Pro and Molmo-72B while also producing segmentations.

Video PCS (Table 5). SA-Co/VEval cgF${}_1$ / pHOTA: SA-V 30.3 / 58.0, YT-Temporal-1B 50.8 / 69.9, SmartGlasses 36.4 / 63.6 — above 80% of human pHOTA, and far above decoupled baselines (“SAM 3 Detector + T-by-D” gets 25.7 / 55.7; GLEE and LLMDet+SAM 3 tracker collapse on large NP vocabularies). Public benchmarks: LVIS 36.3 test mAP, BURST 44.5 test HOTA, YTVIS21 57.4 val mAP, OVIS 60.5 val mAP.

PVS is not sacrificed. VOS (Table 6): MOSEv1 78.4, DAVIS17 92.2, LVOSv2 88.5, SA-V val 83.5, SA-V test 84.4, YTVOS19 $\mathcal{G}$ 89.7, MOSEv2 60.3 (+6.5 over the prior best). Interactive image segmentation on the SA-37 benchmark (Table 7): 1-click 66.1, 3-click 81.3, 5-click 85.1 mIoU vs SAM 2.1 L’s 64.4 / 80.3 / 84.3 — but at 43.5 FPS vs 93.0, the price of the larger, semantically capable backbone.

SAM 3 Agent. An MLLM proposes NP queries, calls SAM 3 as a tool and iterates on the returned masks. Zero-shot, this beats systems fine-tuned on referring/reasoning segmentation data: ReasonSeg test gIoU 74.0 (Gemini 2.5 Pro) vs 65.0 prior state-of-the-art, val 77.0; OmniLabel descr AP 45.3 vs 36.5. It works across Qwen2.5-VL 7B/72B, Llama 4 Maverick and Gemini 2.5 Pro with the same system prompts.

Ablations

• Presence head: 50.7 → 52.2 cgF${}_1$, IL_MCC 0.77 → 0.82.
• Hard negatives per image 0 → 5 → 15 → 30: cgF${}_1$ 28.3 → 39.4 → 41.8 → 43.0, IL_MCC 0.44 → 0.62 → 0.67 → 0.68. The gain is almost entirely image-level recognition.
• Training data: EXT alone 23.7 → +SYN 32.8 (+8.8) → +HQ 47.4 (+14.6). Notably EXT alone already matches the best external models, i.e. the architecture itself is strong.
• Data scaling: SA-Co/HQ scales smoothly (23.7 → 34.0 → 37.3 → 40.0 → 42.2 → 45.5 cgF${}_1$ at 0 / 1 / 4 / 10 / 20 / 100% HQ), and so does SA-Co/SYN.
• AI verifiers on top of SAM 3: replacing the presence score with the EV verifier’s score gives +7.2 cgF${}_1$ (54.0 → 61.2); adding MV mask filtering another +1.1 (→ 62.3) against human 72.8 — the verifier system closes nearly half the model–human gap.
• Automatic domain adaptation: holding out “Food & drink”, synthetic data generated by SAM 3 + AI verifiers (SA-Co/SYN-Food) scales almost as well as human-annotated data (SA-Co/HQ-Food) and far above raw pseudo-labels, with no human annotation — provided high-quality general-domain pre-training data is mixed in 1:1 during fine-tuning.
• Video data: adding SA-Co/VIDEO masklets to Stage-4 training improves $\mathcal{J}\&\mathcal{F}$ on SA-Co/VEval (80.7 → 81.4 YT-1B, 84.9 → 86.5 SA-V) and on public SA-V val (77.0 → 77.4).

Limitations

• Zero-shot generalization to fine-grained out-of-domain concepts (aircraft types, medical terms) and niche visual domains (thermal imagery) is weak. Concept generalization is inherently harder than the class-agnostic generalization that made SAM 1/2 work everywhere zero-shot; the mitigation is small-scale fine-tuning or synthetic domain adaptation, both of which require extra training.
• Only simple noun phrases. No multi-attribute queries beyond one or two attributes, no long phrases, no referring expressions — those require the MLLM-driven SAM 3 Agent.
• Inference cost scales linearly with the number of tracked objects, since every object gets its own SAM 2-style masklet; real-time on a single H200 means ~5 concurrent objects, otherwise multi-GPU parallelization.
• No shared object-level context across tracked objects, so multi-object ambiguities in crowded scenes cannot be resolved jointly; a shared global memory is proposed as future work.
• Hard mode-switch between concept mode and instance mode: refining one instance without disturbing the rest of the concept’s instances requires leaving PCS for PVS, rather than interleaving them.

See also

• SAM — the promptable-segmentation origin; SAM 1 is still the browser-tool annotators use for manual mask correction in the SAM 3 data engine
• SAM2 — the video predecessor whose tracker, memory bank and occlusion head SAM 3 reuses wholesale
• MobileSAM — the opposite direction: distilling SAM’s encoder down instead of scaling capability up
• DETR — the detector’s lineage; SAM 3 stacks iterative box refinement, look-forward-twice, hybrid matching and DAC-DETR on top of it
• DINO — the strong-DETR recipe (query selection, look-forward-twice) SAM 3’s detector inherits
• Implementing Research Papers