SAM2

SAM 2 (Segment Anything in Images and Videos)

SAM 2 is a unified promptable segmentation model for images and videos.

• Paper: Ravi, Gabeur, Hu, Hu, Ryali, Ma, Khedr, Rädle, Rolland, Gustafson, Mintun, Pan, Alwala, Carion, Wu, Girshick, Dollár, Feichtenhofer (Meta FAIR), SAM 2: Segment Anything in Images and Videos, 2024, arXiv:2408.00714.
• Core idea: generalizes SAM to the temporal domain by treating an image as a single-frame video, adding a streaming memory so that a prompt on one frame produces a segmentation on every frame.
• Task and data: alongside it come the PVS task and the SA-V dataset (50.9K videos, 642.6K masklets, 35.5M masks, 53× more masks than any prior video segmentation dataset).
• Headline results: better video segmentation accuracy with 3× fewer interactions than prior approaches; on images it is both more accurate and 6× faster than the original SAM.

The headline results: better video segmentation accuracy with 3× fewer interactions than prior approaches, and on images it is both more accurate and 6× faster than the original SAM.

The PVS task

Promptable Visual Segmentation extends SAM’s promptable segmentation from static images to video. Prompts — positive/negative clicks, boxes, or masks — can be given on any frame, either to define an object or to refine a model prediction. On receiving a prompt the model must immediately return a valid mask on that frame, then propagate it across the whole video to produce a masklet (the spatio-temporal mask). Further prompts on any frame refine the masklet.

Both predecessors are special cases: SA is PVS with the video reduced to one frame, and semi-supervised VOS is PVS restricted to a mask prompt on the first frame only.

The practical payoff of memory is shown below — when the model loses the object, a single click on a later frame recovers it, whereas a decoupled SAM + tracker would need the object re-annotated from scratch because tracking restarts from that frame.

Architecture

Frames are consumed one at a time in a streaming fashion, so arbitrarily long videos can be processed in real time. The prediction for the current frame is conditioned on the current prompt and/or on memories of previously observed frames — which may also come “from the future” relative to the current frame. When the memory is empty, SAM 2 behaves exactly like SAM.

Image encoder

An MAE pre-trained Hiera encoder, run once per frame for the entire interaction, producing unconditioned feature embeddings. Being hierarchical, it supplies multiscale features: an FPN fuses the stride-16 and stride-32 features of Stages 3 and 4 into the frame embedding, while the stride-4 and stride-8 features from Stages 1 and 2 bypass memory attention entirely and are injected into the mask decoder’s upsampling layers for high-resolution detail.

Positional encoding is absolute and windowed; unlike SAM, all relative positional bias (RPB) is removed from the image encoder, which enables FlashAttention-2 and gives a large speed gain at 1024² with no measurable accuracy cost.

Memory attention

$L$ transformer blocks (default $L=4$). The first takes the current frame’s image encoding; each block does self-attention, then cross-attention to the memory bank — both spatial memories of prompted/unprompted frames and object pointers — then an MLP. Attention is vanilla so it can use efficient kernels (FlashAttention-2). Positional encoding here is 2d spatial RoPE in self- and cross-attention; object pointer tokens are excluded from RoPE since they have no spatial correspondence.

Prompt encoder and mask decoder

The prompt encoder is identical to SAM’s. The decoder keeps SAM’s stacked “two-way” blocks that update prompt and frame embeddings jointly, plus SAM’s multi-mask output for ambiguity — but ambiguity in video can extend across frames, so the model predicts multiple masks at each step and, when no follow-up prompt resolves it, propagates only the mask with the highest predicted IoU.

Two additions over SAM:

• Occlusion head. Unlike images, where a positive prompt always implies a valid object, in PVS the object may be absent on some frames (e.g. occlusion). An extra output token plus MLP head predicts an occlusion score — whether the object of interest is present in the current frame. A learned occlusion embedding is added to the memory features of frames predicted to be invisible.
• Object pointer. The mask token corresponding to the output mask doubles as an object pointer for the frame, a lightweight high-level semantic vector stored in the memory bank.

Memory encoder and memory bank

The memory encoder needs no separate image encoder: it downsamples the predicted output mask with a convolutional module, sums it element-wise with the unconditioned frame embedding from the Hiera encoder, and fuses with light convolutional layers — so memories inherit the image encoder’s representational strength for free as the encoder scales.

The memory bank is two FIFO queues: up to $N$ recent frames, and up to $M$ prompted frames. (In VOS, where the first-frame mask is the only prompt, this means the first frame’s memory is always retained alongside the $N$ most recent unprompted frames.) Memory features are projected to 64 channels for storage, and the 256-dim object pointer is split into 4 tokens of 64-dim for cross-attention.

Temporal position embeddings are added to the memories of the $N$ recent frames — but deliberately not to prompted frames, since the training signal there is sparse and prompted frames at inference may come from a very different temporal range than seen during training.

Multiple objects are handled by running inference per object: only the image encoder features are shared, while memory bank and mask decoder run separately for each object.

Training

Training is joint on image and video data, alternating between the two sources with sampling probability proportional to each dataset’s size, and a different batch size per source.

The video task simulates interactivity: sample 8-frame sequences, randomly select up to 2 frames (always including the first) to receive prompts, and probabilistically supply corrective clicks sampled from the ground-truth masklet and the model’s own predictions. The initial prompt is the ground-truth mask with p = 0.5, a positive click with p = 0.25, or a bounding box with p = 0.25. At most 3 masklets per 8-frame sequence; temporal order is reversed with p = 0.5 to encourage bi-directional propagation; with p = 0.1 corrective clicks are sampled from ground truth irrespective of the model prediction, for extra flexibility.

A mosaic augmentation (p = 0.1) tiles the same video into a 2×2 grid and targets a masklet in one quadrant, forcing the model to use motion and temporal continuity rather than appearance to disambiguate identical-looking counterparts — and, since each quadrant is half-size, also helping on small objects.

A final 16-frame fine-tuning stage addresses the gap between 8-frame training and full-length inference: 50k iterations (⅓ of the original schedule) at half the learning rate, on the top 50% most-edited (i.e. most challenging) masklets, with the image encoder frozen to fit 16 frames into 80 GB A100s.

Losses

Mask prediction uses a linear combination of focal and dice loss, IoU prediction uses MAE (${\ell }_1$), and object (occlusion) prediction uses cross-entropy, in a ratio of 20 : 1 : 1 : 1. As in SAM, for multi-mask predictions only the mask with the lowest segmentation loss is supervised — but unlike SAM, the IoU predictions of all masks are supervised (with a sigmoid on the IoU logits restricting them to $[0,1]$) to better learn when a mask might be bad. If the ground truth has no mask on a frame, no mask output is supervised, but the occlusion head always is.

Hyperparameters

From Table 12. Values in parentheses vary with image encoder size (T, S, B+, L).

config	pre-training	full training
Data	SA-1B	SA-1B, Internal, SA-V
Steps	~90k	~150k
Resolution	1024	1024
Precision	bfloat16	bfloat16
Optimizer	AdamW (${\beta }_1,{\beta }_2=0.9,0.999$)	AdamW (${\beta }_1,{\beta }_2=0.9,0.999$)
Gradient clipping	${\ell }_2$, max 0.1	${\ell }_2$, max 0.1
Weight decay	0.1	0.1
Learning rate	4e-4	img. enc. 6e-5, other 3.0e-4
LR schedule	reciprocal sqrt, timescale 1000	cosine
Warmup	linear, 1k iters	linear, 7.5k iters
Cooldown	linear, 5k iters	—
Layer-wise decay	0.8 (T, S), 0.9 (B+), 0.925 (L)	0.8 (T, S), 0.9 (B+), 0.925 (L)
Batch size	256	256
Drop path	0.1 (T, S), 0.2 (B+), 0.3 (L)	0.1 (T, S), 0.2 (B+), 0.3 (L)
Mask losses (weight)	focal (20), dice (1)	focal (20), dice (1)
IoU loss (weight)	${\ell }_1$ (1)	${\ell }_1$ (1)
Occlusion loss (weight)	—	cross-entropy (1)
Max masks per frame	64	image 64, video 3
# correction points	7	7
Global attn. blocks	5-7-9 (T), 7-10-13 (S), 12-16-20 (B+), 23-33-43 (L)	same

Image augmentation is horizontal flip + resize to 1024 square. Video augmentation additionally uses affine (degrees 25, shear 20), color jitter (brightness 0.1, contrast 0.05, saturation 0.03), grayscale (0.05), per-frame color jitter (brightness 0.1, contrast 0.05, saturation 0.05) and mosaic-2×2 (0.1).

Other defaults: memory attention $L=4$ layers, $N=6$ past frames of memory, 64 memory channels, 8-frame training sequences, 1024² resolution. Pre-training filters masks covering >90% of the image and restricts to 64 randomly sampled masks per image; SAM’s two extra no-prompt refinement iterations are not used, and 7 correction clicks are used instead of SAM’s 8.

The full training data mixture is ~15.2% SA-1B, ~70% SA-V, ~14.8% Internal (or with the open-source datasets included: ~1.3% DAVIS, ~9.4% MOSE, ~9.2% YouTubeVOS, ~15.5% SA-1B, ~49.5% SA-V, ~15.1% Internal). Only manually annotated masklets are used from SA-V during training; the automatic ones are not needed for the reported results.

Data engine

Three phases, each with more model assistance, and no semantic constraints on what gets annotated (whole objects and parts):

	Model in the loop	Time / frame	Edited frames	Clicks / clicked frame
Phase 1	SAM only	37.8 s	100.00%	4.80
Phase 2	SAM + SAM 2 Mask	7.4 s	23.25%	3.61
Phase 3	SAM 2	4.5 s	19.04%	2.68

1. SAM per frame. Annotate every frame at 6 FPS with SAM plus brush/eraser. No tracking, so every frame is annotated from scratch — slow, but yields high-quality spatial annotations. 16K masklets across 1.4K videos. This protocol is also used to annotate the SA-V val and test sets, to avoid SAM 2 biases at evaluation time.
2. SAM + SAM 2 Mask. “SAM 2 Mask” accepts masks only as prompts: annotate frame 1 spatially with SAM, then temporally propagate; correct any bad intermediate frame by re-annotating from scratch and re-propagating. 5.1× speedup; 63.5K masklets; SAM 2 Mask retrained twice in-loop.
3. SAM 2. The full model, accepting points and masks and carrying memory, so annotators only supply occasional refinement clicks instead of annotating from scratch. 8.4× speedup over Phase 1 at comparable quality; retrained 5 times in-loop; 197.0K masklets.

Phase 3 is not just faster but better: Phase 1 Mask Alignment (fraction of masks with IoU > 0.75 against the Phase 1 reference) is 89.1% overall and 100.0% on large objects, vs. 86.4% / 97.9% for Phase 2.

A quality verification step has separate annotators rate each masklet “satisfactory” or “unsatisfactory”; unsatisfactory ones return to the pipeline, and masklets tracking objects that are not well defined are rejected outright.

Auto masklet generation prompts SAM 2 with a regular grid on the first frame (32×32 on the full frame, plus 16×16 on 4 zoomed crops from a 2×2 window and 4×4 on 16 crops from a 4×4 window). Auto masklets rated satisfactory go straight into SA-V; ones rated unsatisfactory are exactly the model’s failure cases, and get sent to annotators for Phase 3 refinement. Post-processing removes components under 200 px and fills holes under 200 px.

Each data engine phase measurably helps ($\mathcal{J}\&\mathcal{F}$ on SA-V val / 9 zero-shot): VOS + SA-1B 50.0 / 62.5 → +Phase 1 53.0 / 66.9 → +Phase 2 58.8 / 70.9 → +Phase 3 62.5 / 71.2 → +Auto 63.2 / 71.5.

SA-V dataset

50.9K crowdworker-recorded videos (54% indoor / 46% outdoor, avg. 13.8 s, 240p–4K, 1401×1037 on average, 196 hours, 4.2M frames) across 47 countries, with 190.9K manual + 451.7K automatic masklets = 642.6K masklets / 35.5M masks.

	videos	Duration	masklets	masks	Disapp. rate
DAVIS 2017	0.2K	0.1 hr	0.4K	27.1K	16.1%
YouTube-VOS	4.5K	5.6 hr	8.6K	197.3K	13.0%
MOSE	2.1K	7.4 hr	5.2K	431.7K	41.5%
BURST	2.9K	28.9 hr	16.1K	600.2K	37.7%
SA-V Manual	50.9K	196.0 hr	190.9K	10.0M	42.5%
SA-V Manual+Auto	50.9K	196.0 hr	642.6K	35.5M	27.7%

That is 53× more masks than the largest existing VOS dataset (15× without the automatic annotations). Over 88% of SA-V masks have normalized area below 0.1 — the dataset is dominated by small objects and parts. The disappearance rate of 42.5% (masklets that vanish for at least one frame and reappear) is high, i.e. deliberately hard. Splits are by video author and geographic location to minimize overlap of similar objects; val (155 videos / 293 masklets) and test (150 / 278) target challenging fast-moving, occluded and re-appearing objects.

Results

Promptable video segmentation. Across 9 zero-shot datasets under both offline and online evaluation with 3 clicks per frame, SAM 2 dominates SAM+XMem++ and SAM+Cutie at every number of interacted frames, achieving better accuracy with >3× fewer interactions.

Semi-supervised VOS (prompt on first frame only, 17 zero-shot datasets):

Method	1-click	3-click	5-click	box	GT mask
SAM+XMem++	56.9	68.4	70.6	67.6	72.7
SAM+Cutie	56.7	70.1	72.2	69.4	74.1
SAM 2	64.7	75.3	77.6	74.4	79.3

Against VOS state-of-the-art with ground-truth first-frame masks, SAM 2 (Hiera-L) reaches 77.9 MOSE val, 90.7 DAVIS 2017 val, 78.0 LVOS val, 76.8 SA-V val, 78.4 SA-V test, 89.3 YTVOS 2019 val — with the gap on SA-V val/test being by far the largest, showing how much prior methods lack a true “segment anything” capability. Hiera-B+ and Hiera-L run at 43.8 and 30.2 FPS on one A100.

Image segmentation (SA task, 37 zero-shot datasets, 1-click mIoU):

Model	Data	SA-23 All	SA-23 Image	SA-23 Video	14 new Video	FPS
SAM (ViT-H)	SA-1B	58.1	60.8	54.5	59.1	21.7
SAM 2	SA-1B	58.9	60.8	56.4	56.6	130.1
SAM 2	our mix	61.9	63.3	60.1	69.6	130.1

Trained on SA-1B alone, SAM 2 already beats SAM while being 6× faster — attributable to the smaller but more effective Hiera encoder. Adding the video data mix lifts SA-23 to 61.9 and gives exceptional gains on the video-derived benchmarks (69.6 vs. 59.1 on the 14 new video datasets).

Ablations

• Data mix. Training on VOS datasets alone scores well in-domain (MOSE dev) but poorly everywhere else; adding data-engine data gives +12.1% average on the 9 zero-shot datasets. Adding SA-1B improves image segmentation without degrading VOS. Best results mix all three: VOS + SA-1B + SA-V.
• Data quantity. Accuracy follows a consistent power law in the number of masklets across SA-V val, zero-shot, and MOSE dev.

• Data quality. Selecting 50k masklets by most edited frames beats random 50k sampling (66.2 vs. 63.7 SA-V val) using just 25% of the data — the edit count is a good proxy for difficulty — though the full 190k is still better (69.9).
• Capacity. Resolution 512 → 1024 helps substantially; 8 frames balances speed and accuracy; $N=6$ memories is the sweet spot; reducing memory channels 256 → 64 costs little while cutting storage 4×. Scaling the image encoder helps both image and video metrics, scaling memory attention only helps video.
• Memory design. Feeding memories through a GRU before the memory bank gives no improvement — directly storing the features is simpler and more efficient. Cross-attending to object pointers doesn’t move the 9-dataset zero-shot average but clearly helps SA-V val (68.3 vs. 64.5) and long-video LVOSv2 (71.6 vs. 67.0), so it is kept.

Limitations

• Can fail across shot changes, and may lose or confuse objects in crowded scenes, after long occlusions, or in extended videos — mitigated, but not solved, by the ability to prompt any frame for recovery.
• Struggles with very thin or fine details, especially on fast-moving objects.
• Nearby objects of similar appearance (e.g. multiple identical juggling balls) cause confusion; explicit motion modelling could help.
• Multiple objects are processed independently, sharing only per-frame image embeddings — no inter-object communication, which is simple but leaves efficiency on the table.
• The data engine still relies on human annotators for masklet verification and frame selection.

Fairness

Evaluated on the people category of Ego-Exo4D, $\mathcal{J}\&\mathcal{F}$ gaps across perceived gender are 81.9 (male) vs. 75.1 (female) at 1 click, narrowing to 95.1 / 94.1 at 3 clicks and 95.9 / 95.2 with a ground-truth mask. Much of the 1-click gap is prompt ambiguity — the model often segments a part rather than the person; restricted to clips where the person is correctly segmented, the 1-click gap shrinks to 94.3 / 92.7. Across age groups (18–26, 26–50, 50+) there is little variance.

See also

• SAM — the image-only predecessor SAM 2 generalizes
• DETR — the Transformer decoder lineage behind both mask decoders
• Implementing Research Papers