SigLIP2

SigLIP 2 is a family of multilingual vision-language encoders that keeps SigLIP’s architecture and sigmoid image-text loss unchanged but replaces its single-loss training recipe with a combined one.

• Paper: Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner and Xiaohua Zhai (Google DeepMind), SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, arXiv:2502.14786, Feb. 2025.
• Drop-in replacement: the architecture and sigmoid image-text loss are unchanged, so existing users can swap in the new weights and multilingual tokenizer.
• Core idea: a combined recipe that folds in captioning-based pretraining (LocCa), self-distillation and masked prediction (SILC/TIPS), and active-data-curation distillation for the smallest models.
• Headline result: outperforms SigLIP at every model scale on zero-shot classification, retrieval, and VLM transfer.
• New capabilities SigLIP lacked outright: strong open-vocabulary localization; dense (per-patch) semantic features usable for segmentation and depth; native aspect-ratio/resolution support (NaFlex); markedly reduced representation bias thanks to de-biased multilingual training data.
• Checkpoints: four sizes, ViT-B (86M), L (303M), So400m (400M), g (1B), trained on the 10-billion-image, 109-language WebLI dataset.

Fig. 1: SigLIP 2 adds the captioning-based decoder loss from LocCa and the self-distillation/masked-prediction losses from SILC/TIPS (last 20% of training) on top of the SigLIP (v1) sigmoid loss. Dashed purple box = the original SigLIP subgraph.

Motivation: what SigLIP v1 leaves on the table

SigLIP replaced CLIP’s softmax contrastive loss with a per-pair sigmoid (binary classification) loss, which decouples the loss from the batch’s global normalization and scales more efficiently to large batches. But it is still, fundamentally, a single-objective, image-level, English-only recipe, and that leaves three gaps that SigLIP 2 sets out to close:

• Semantic understanding is capped by contrastive-only supervision. Purely contrastive objectives learn a good global image-text alignment but comparatively weak dense/local semantics, because the loss only ever supervises a single pooled embedding per image. Prior work (SILC, TIPS, DINO) shows that adding self-distillation and masked-prediction objectives — which force local patch features to be predictive of a global teacher view — substantially improves the quality of unpooled, per-patch representations.
• No localization or OCR signal. SigLIP has no notion of where in the image something is, or of reading text in the image, because it never predicts free-form text conditioned on spatial structure. LocCa-style captioning, dense captioning, and referring-expression decoding inject exactly this signal via an auxiliary decoder, and were shown (independently, in Image Captioners Are Scalable Vision Learners Too and LocCa) to improve OCR and localization while still producing good global representations.
• Fixed square resolution and English-only data. SigLIP checkpoints are trained and served at a single fixed, aspect-ratio-distorting resolution, and predominantly on English web text, which both limits document/OCR-style use cases (aspect-sensitive layouts) and leaves the model with weak multilingual understanding and measurable gender/association biases.

SigLIP 2 addresses all three without discarding backward compatibility: the base sigmoid loss and ViT/text-encoder architecture from SigLIP are kept exactly, so the new losses are strictly additive during training and disappear at inference (the decoder and self-distillation heads are training-only scaffolding, not shipped in the released encoder).

Architecture

Unchanged from SigLIP by design: a standard ViT image tower and a matched-size text tower (except the g-sized vision encoder, which is paired with a smaller So400m-sized text encoder), vision and text representations pooled with a MAP head (multi-head attention pooling) rather than a CLS token, text sequence length 64, and the multilingual Gemma tokenizer (256k vocabulary, lower-cased before tokenization) replacing SigLIP’s English-only tokenizer. Patch size is 16 and the base image resolution is 256 (256 image tokens) for all released fixed-resolution checkpoints. Training data is WebLI: 10B images / 12B alt-texts across 109 languages, mixed 90% English-sourced / 10% non-English-sourced web pages (following prior findings that this ratio best balances English and multilingual quality), with the de-biasing filtering techniques from prior SigLIP-bias work applied to mitigate gender-representation and gender-occupation association biases.

Training recipe: combining four objectives in stages

Rather than training all losses jointly from scratch (computationally and memory-heavy), SigLIP 2 applies them in stages on top of a shared image/text encoder pair.

Stage 1 — sigmoid loss + LocCa decoder (0–80% of training)

The base loss is unchanged SigLIP: every image embedding in a mini-batch is paired with every text embedding, and each pair is scored as a binary matching/non-matching classification via a sigmoid (logistic-regression-style) loss rather than a batch-softmax contrastive loss.

On top of this, a standard transformer decoder is attached via cross-attention to the unpooled vision-encoder output (before the MAP head), following the shapes of the text encoder but with added cross-attention layers and half as many layers. This decoder is trained for three captioning-style targets per example (three forward passes): (1) standard image captioning, (2) dense/region captioning — predicting a caption for a given bounding box, and (3) automatic referring-expression prediction — predicting bounding-box coordinates for a phrase describing a specific region. Region-caption pairs are mined automatically by extracting n-grams (or, for consistency with prior work, a fixed object-category vocabulary) from image alt-text and applying open-vocabulary detection to localize them. The captioning target uses parallel prediction (all tokens decoded in parallel from mask tokens, no causal masking) with 50% probability, to cut decoding cost; a chunked decoder-loss implementation keeps memory tractable against the large 256k vocabulary. The sigmoid loss and the LocCa decoder loss are combined with equal weight.

Stage 2 — add self-distillation and masked prediction (last 20% of training)

Following SILC and TIPS, two further terms are added for the final 20% of training to improve local/dense semantics:

1. Local-to-global consistency (self-distillation) loss. The image encoder is duplicated into a student and a teacher; the teacher’s weights are an exponential moving average (EMA) of the student’s over previous iterations, and the teacher’s gradient path is stopped. The teacher sees the full (global) image view; the student sees a local (partial) crop. The student is trained to match the teacher’s representation — computed in a separate high-dimensional feature space via an auxiliary MLP head — for the same underlying image. The setup uses 1 global (teacher) view and 8 local (student) views per image, with augmentations, loss, and other hyperparameters following SILC.
2. Masked prediction loss. 50% of the embedded image patches on the student side are replaced with mask tokens; the student is trained to match the teacher’s per-patch (not pooled) features at the masked locations. Both student and teacher otherwise see the same global view (up to student-side masking). This is the same consistency loss as term 1 but applied per-patch rather than to the pooled representation.

Both new losses are computed on separately augmented views of the image (not the original), so that this auxiliary self-supervision cannot corrupt the image-text alignment learned by the sigmoid/LocCa terms on the original image — a precaution recommended by the SILC authors. The teacher is initialized from the student’s weights at the 80% mark, with its extra parameters (heads, mask token, optimizer state) initialized randomly.

Weighting: within the new pair, the consistency (term 1) and masked-prediction (term 2) losses are weighted 1 and 0.25 respectively. The combined new-loss group is then weighted against the stage-1 losses by an additional factor of 0.25, 0.5, 1.0, 0.5 for the B, L, So400m and g model sizes respectively — i.e. the dense-feature objective is weighted differently per model size to balance semantic/global quality against dense-task quality.

Stage 2b — active-data-curation distillation for the smallest models (ViT-B/16, ViT-B/32 only)

To specifically improve the smallest checkpoints, SigLIP 2 fine-tunes ViT-B/16 and ViT-B/32 for an additional 4B examples with implicit distillation via active data curation (ACID): a strong SigLIP 2 So400m teacher and the current learner jointly score candidate examples from a larger super-batch by their “learnability,” and an optimal batch (size 32k, drawn from a 64k super-batch, filtering ratio 0.5, or 0.75 for B/32) is selected each step to balance data-quality gains against training compute. Unlike the prior ACED method (which combines ACID with an additional, explicit softmax-distillation loss against a second teacher trained on more diverse data), SigLIP 2 uses a single teacher — the So400m model itself fine-tuned for 1B examples on a curated high-quality dataset — and finds this recovers ACED’s benefits without the extra distillation loss or second teacher, at lower compute. During this fine-tuning stage the learning rate is lowered to $10^{-5}$, weight decay is removed, and only the sigmoid image-text loss is used (no decoder/self-distillation losses).

NaFlex: native aspect ratio and variable resolution

Separately from the fixed-resolution checkpoints (obtained by resuming a 256-sequence-length checkpoint at 95% of training and resizing the positional embedding and, in some cases, the patch embedding via pseudoinverse resizing), SigLIP 2 also trains a NaFlex variant that combines two prior ideas into one checkpoint:

• FlexiViT: a single model that supports multiple predefined patch sizes / sequence lengths.
• NaViT: processing images at their native aspect ratio rather than a fixed square resize.

Given a patch size and target sequence length, NaFlex resizes an input image so that height and width are multiples of the patch size while (1) minimizing aspect-ratio distortion and (2) producing a sequence length at most the target — bounding the resize distortion to at most $(\text{patch\_size}-1)/\text{width}$ and $(\text{patch\_size}-1)/\text{height}$ per axis. The learned positional embedding (defined for a $16\times 16$ grid) is bilinearly resized (with anti-aliasing) to the resized image’s non-square patch grid; attention layers (including the MAP head) mask out padding tokens when the actual sequence is shorter than the target. NaFlex checkpoints are obtained by resuming the 90%-trained default (256-sequence, non-aspect-preserving) checkpoint, then training on aspect-preserving resizes with a sequence length uniformly sampled per mini-batch from $\{128,256,576,784,1024\}$, stretching the last-10%-of-training LR schedule by $3.75\times$ so each resolution sees enough examples (and halving batch size / doubling steps at the largest sequence length to fit memory). NaFlex does not apply the self-distillation/masked-prediction losses (Stage 2), to keep implementation and compute manageable.

Hyperparameters

Transcribed from §2.1–2.5.

Base pretraining (Stage 1 + Stage 2)

Optimizer	Adam, LR $10^{-3}$, decoupled weight decay $10^{-4}$
Gradient clipping	max norm 1
Batch size	32,768
LR schedule	cosine, 20k warm-up steps
Total examples seen	40B
Hardware	up to 2048 TPUv5e chips, FSDP (fully-sharded data-parallel)
Patch size / resolution (fixed-res. checkpoints)	16 / 256px (256 tokens)
Text sequence length	64
Tokenizer	multilingual Gemma tokenizer, vocab 256k, lower-cased
Data	WebLI, 10B images / 12B alt-texts, 109 languages, 90% English-source / 10% non-English-source
Stage-1 loss weights	sigmoid : LocCa decoder = 1 : 1
Stage 2 start	80% of training
Stage-2 views	1 global (teacher) + 8 local (student)
Stage-2 masking ratio	50% of student patches
Stage-2 term weights (consistency : masked-pred.)	1 : 0.25
Stage-2 group weight vs. Stage-1 (B / L / So400m / g)	0.25 / 0.5 / 1.0 / 0.5

NaFlex adaptation

Start point	90%-trained default checkpoint
Sequence lengths trained	$\{128,256,576,784,1024\}$, uniformly sampled per mini-batch
LR-schedule stretch (last 10%)	$3.75\times$
Largest-sequence adjustment	batch size halved, steps doubled
Self-distillation / masked prediction	not applied

Active-data-curation distillation (ViT-B/16, ViT-B/32 only)

Additional examples	4B
LR	$10^{-5}$ (lowered), weight decay removed
Teacher	SigLIP 2 So400m, fine-tuned 1B examples on a curated high-quality dataset
Method	ACID (single teacher), not ACED (which uses 2 teachers + explicit distillation loss)
Batch / super-batch size	32k / 64k
Filtering ratio	0.5 (B/16), 0.75 (B/32)
Losses active	sigmoid image-text loss only

Released checkpoint sizes: ViT-B (86M), L (303M), So400m (400M), g (1B parameters); g-sized vision encoder is paired with a So400m-sized text encoder.

Results

Zero-shot classification & retrieval. Across all four model sizes SigLIP 2 outperforms SigLIP and other open baselines (OpenCLIP, MetaCLIP, EVA-CLIP, DFN) on ImageNet-1k (val/v2/ReaL/ObjectNet), COCO, Flickr, and the 36-language Crossmodal-3600 retrieval benchmark — despite also supporting many more languages than most baselines. E.g. at So400m/14, 384px: ImageNet-1k val 83.1 (SigLIP 2) vs. 83.2 (SigLIP, comparable) but COCO T→I 55.8 vs. 50.8 and Flickr T→I 85.7 vs. 80.5; gains are largest for the small B-sized models thanks to the ACID distillation stage. On multilingual retrieval (Crossmodal-3600), SigLIP 2 closes most of the gap to mSigLIP (a SigLIP variant trained purely on multilingual data) while remaining far better on English-centric benchmarks — i.e. it gets multilinguality largely “for free.”

Vision encoder for VLMs. Plugging SigLIP 2 into a Gemma 2-based VLM (a PaliGemma-style stage-1/stage-3 recipe: frozen vision encoder + LLM co-trained on captioning/OCR/detection/segmentation tasks, then fine-tuned per downstream dataset) beats SigLIP and AIMv2 as the vision backbone across essentially the entire evaluated suite (AI2D, DocVQA, InfoVQA, OCR-VQA, RefCOCO/+/g, ScienceQA, TextVQA, TallyQA, XM3600, etc.), with the largest jumps on OCR/document tasks (DocVQA, InfoVQA, OCR-VQA) — attributable to the LocCa decoder pretraining.

Dense prediction (frozen features). Probing frozen SigLIP 2 features (So/14, linear or DPT-decoder probe) against SigLIP, CLIP, and OpenCLIP:

Model	ViT	Res.	Seg. PASCAL (mIoU↑)	Seg. ADE20k (mIoU↑)	Depth NYUv2 (RMSE↓)	Depth NAVI (RMSE↓)
CLIP	L/14	224	74.5	39.0	0.553	0.073
OpenCLIP	G/14	224	71.4	39.3	0.541	–
SigLIP	So/14	224	72.0	37.6	0.576	0.083
SigLIP 2	So/14	224	77.1	41.8	0.493	0.067
SigLIP	So/14	384	73.8	40.8	0.563	0.069
SigLIP 2	So/14	384	78.1	45.4	0.466	0.064

This is the clearest demonstration that Stage-2 self-distillation/masked prediction achieves its goal: dense per-patch features improve substantially over the sigmoid-only SigLIP baseline, at both resolutions.

Open-vocabulary segmentation (Cat-Seg framework, trained on COCO-Stuff, evaluated zero-shot on ADE20k/Pascal Context/Pascal VOC vocabularies): SigLIP 2 L/16 improves over SigLIP L/16 on every vocabulary and even surpasses the much larger OpenCLIP G/14 on most of them (e.g. VOC-21: 82.3 vs. 81.1 for SigLIP, 81.4 for OpenCLIP G/14).

Localization: referring-expression comprehension (RefCOCO/+/g, 6-layer decoder trained from scratch on frozen features): SigLIP 2 beats SigLIP and CLIP by a wide margin at every size and resolution (e.g. B/16 @576: RefCOCO testA 92.74 vs. 72.94 for SigLIP). It is only outperformed by LocCa itself, which the authors attribute to LocCa training purely on English data, whereas SigLIP 2 is multilingual.

Open-vocabulary detection (OWL-ViT recipe, fine-tuned): SigLIP 2 improves both COCO AP and LIVS AP over SigLIP, with the largest relative gain on LVIS rare categories (APr): e.g. So/14, 40.9 → 42.3.

NaFlex vs. fixed-resolution. On OCR/document-heavy retrieval benchmarks (TextCaps, HierText, SciCap, Screen2Words) NaFlex outperforms the standard square-resize variant, especially at small sequence lengths where aspect-ratio distortion hurts most; on natural-image benchmarks the standard variant is usually slightly ahead (partly because it additionally benefits from the Stage-2 self-distillation losses, which NaFlex skips). NaFlex interpolates well between its trained sequence lengths but does not extrapolate beyond them.

Cultural diversity and fairness. SigLIP 2 improves zero-shot accuracy on geographically-diverse benchmarks (Dollar Street, GeoDE) and 10-shot geolocalization (e.g. GeoDE region accuracy: 36.2% → 44.4% for L/16 @256px). Representation bias (tendency to associate a random object with one gender) drops sharply, e.g. SigLIP L/16 @256px: 35.5% → 7.3% for SigLIP 2 of the same size/resolution; larger models are less biased than smaller ones in both families.

Fig. 3: representation bias (association of a random object with a gender; lower is better). SigLIP 2’s de-biased multilingual training data roughly halves-to-fifths the bias of the matching SigLIP checkpoint at both sizes.

Limitations

• NaFlex trades off dense-feature quality. To keep the variable-resolution adaptation tractable, NaFlex skips the Stage-2 self-distillation/masked-prediction losses entirely, so it forgoes exactly the dense-feature gains that are otherwise SigLIP 2’s headline improvement, and it does not extrapolate to sequence lengths outside its training set.
• LocCa still beats SigLIP 2 on localization, because LocCa is trained on English-only data; SigLIP 2’s multilingual data mixture costs it some English-localization accuracy relative to a hypothetical English-only version of the same recipe.
• Active-data-curation distillation is applied only to the two smallest checkpoints (B/16, B/32); larger models do not get this extra fine-tuning stage.
• The combined recipe is staged, not jointly optimal end-to-end — losses are introduced sequentially (captioning from the start, self-distillation only for the last 20%) for compute/memory reasons, not because that schedule was shown to be optimal.
• Bias mitigation reduces but does not eliminate representation bias or income-level disparity (Dollar Street disparity drops only marginally, Table 9 of the paper), and cultural-diversity gains are concentrated in geolocalization rather than uniformly across all evaluated axes.
• As with SigLIP, the encoder itself is released without the decoder — the LocCa decoder used during pretraining is training-only scaffolding and is not part of the public checkpoints.

See also

• SigLIP — the predecessor whose architecture and sigmoid loss SigLIP 2 keeps unchanged, replacing only the training recipe
• Image Captioners Are Scalable Vision Learners Too — the captioner-as-representation-learner idea that LocCa-style decoder pretraining builds on
• How to train your ViT — ViT training/scaling practices underlying the architecture and optimizer choices
• PaliGemma — the VLM recipe SigLIP 2’s “vision encoder for VLMs” evaluation follows
• DETR — for contrast, an end-to-end detection transformer rather than an open-vocabulary/contrastive localization approach