SigLIP

SigLIP (Sigmoid Loss for Language-Image Pre-training) replaces the softmax-based contrastive objective used by CLIP and ALIGN with a simple pairwise sigmoid loss.

• Paper: Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov and Lucas Beyer (Google DeepMind, Zürich), Sigmoid Loss for Language Image Pre-Training, ICCV 2023, arXiv:2303.15343.
• Core idea: every image-text pair is treated as an independent binary classification problem (“is this pair matched or not?”) instead of requiring a batch-wide normalization.
• Properties: needs no global view of the batch’s pairwise similarity matrix, decouples the loss definition from the batch size, uses less memory, and outperforms the softmax loss at small-to-moderate batch sizes while matching it at large ones.
• Headline results: a SigLiT model reaches 79.7% zero-shot ImageNet accuracy training on just four TPU-v4 chips for one day; a from-scratch SigLIP model reaches 73.4% with 32 chips in five days.
• Used by: the vision encoder trained with this recipe (particularly the Shape-Optimized SoViT-400m variant) is what PaliGemma uses.
• Extended by: the loss and recipe are extended by SigLIP2.

Fig. 1: efficient loss implementation. Three devices, global batch 12. Positives are computed locally (a,b), then negatives are gathered by permuting representations device-to-device (c,d) rather than performing an all-gather — only the highlighted $b\times b$ tile is ever materialized.

Motivation: softmax normalization is a global, expensive dependency

Standard contrastive language-image pre-training (CLIP, ALIGN) aligns an image encoder $f(\cdot )$ and text encoder $g(\cdot )$ by pulling matched pairs $(I_i,T_i)$ together and pushing unmatched pairs $(I_i,T_{j\ne i})$ apart, using a batch-level softmax loss applied twice — once normalizing over images for a fixed text, once over texts for a fixed image:

$$-\frac{1}{2|\mathcal{B}|}\sum _{i=1}^{|\mathcal{B}|}\left(\underset{\text{image}\to \text{text}}{\underbrace{\log \frac{e^{t{\mathbf{x}}_i\cdot {\mathbf{y}}_i}}{{\sum }_{j=1}^{|\mathcal{B}|}{e}^{t{\mathbf{x}}_i\cdot {\mathbf{y}}_j}}}}+\underset{\text{text}\to \text{image}}{\underbrace{\log \frac{e^{t{\mathbf{x}}_i\cdot {\mathbf{y}}_i}}{{\sum }_{j=1}^{|\mathcal{B}|}{e}^{t{\mathbf{x}}_j\cdot {\mathbf{y}}_i}}}}\right)$$

where ${\mathbf{x}}_i=f(I_i)/\Vert f(I_i){\Vert }_2$, ${\mathbf{y}}_i=g(T_i)/\Vert g(T_i){\Vert }_2$, and $t=\exp (t^{\prime })$ is a learnable temperature. Computing this exactly under data parallelism requires an all-gather of every embedding across devices and materializing a $|\mathcal{B}|\times |\mathcal{B}|$ pairwise similarity matrix, which is both memory-hungry and numerically delicate — a naive softmax must be stabilized by first finding the row-wise maximum, which itself requires an extra pass over the full (gathered) batch. This global normalization is exactly what ties softmax-style contrastive learning to expensive infrastructure and very large batch sizes (CLIP: ~32k, BASIC/LAION-style studies pushing 16k–160k).

Sigmoid loss for language-image pre-training

SigLIP reformulates the problem as independent binary classification over all pair combinations in the batch — positive label for matched $(I_i,T_i)$, negative for all $(I_i,T_{j\ne i})$:

$$-\frac{1}{|\mathcal{B}|}\sum _{i=1}^{|\mathcal{B}|}\sum _{j=1}^{|\mathcal{B}|}\underset{{\mathcal{L}}_{ij}}{\underbrace{\log \frac{1}{1+e^{z_{ij}(-t{\mathbf{x}}_i\cdot {\mathbf{y}}_j+b)}}}}$$

where $z_{ij}=1$ if $(I_i,T_j)$ are paired and $-1$ otherwise. Since every term ${\mathcal{L}}_{ij}$ is now a fully local, independent computation, the loss requires no operation across the full batch — no all-gather, no global normalization, no numerically-sensitive max-subtraction — which is what makes it simpler to implement in a distributed setting and cheaper in memory.

A learnable bias term $b$ is added alongside the learnable temperature $t=\exp (t^{\prime })$ to counteract the severe class imbalance: for batch size $|\mathcal{B}|$ there are $|\mathcal{B}|$ positive pairs but $|\mathcal{B}{|}^2-|\mathcal{B}|$ negative ones (e.g. 268M negatives for only 16k positives at $|\mathcal{B}|=16$k), which at initialization would otherwise dominate the loss and force large, destabilizing early corrective gradient steps. $b$ and $t^{\prime }$ are initialized to $-10$ and $\log 10$ respectively so training starts close to the expected prior instead of requiring massive over-correction (Table 4 ablation: disabling $b$, or initializing it to 0, both degrade downstream accuracy measurably).

# img_emb : image model embedding [n, dim]
# txt_emb : text model embedding [n, dim]
# t_prime, b : learnable temperature and bias
# n : mini-batch size
t = exp(t_prime)
zimg = l2_normalize(img_emb)
ztxt = l2_normalize(txt_emb)
logits = dot(zimg, ztxt.T) * t + b
labels = 2 * eye(n) - ones(n)   # -1 with diagonal 1
l = -sum(log_sigmoid(labels * logits)) / n

Efficient “chunked” implementation

Because the sigmoid loss decomposes into a sum of independent per-pair terms, a memory- and communication-efficient distributed implementation is possible that avoids materializing the full $|\mathcal{B}|\times |\mathcal{B}|$ matrix at all. With per-device batch $b=|\mathcal{B}|/D$ across $D$ devices, each device first computes the loss for its local positives and $b-1$ local negatives, then representations are permuted (not all-gathered) between neighboring devices one step at a time, accumulating the loss with respect to a fresh $b\times b$ chunk of negatives on each step, until every image has interacted with every text after $D$ steps (Fig. 1). Peak memory is thus $O(b^2)$ rather than $O(|\mathcal{B}{|}^2)$, and the collective permutes are cheaper than the all-gathers a softmax-based loss requires. This chunked scheme is what enables training at batch sizes over one million on comparatively few accelerators.

Model naming: SigLiT vs. SigLIP

The paper studies the sigmoid loss in two settings that reuse CLIP’s naming convention: SigLiT applies it to Locked-image Tuning (LiT) — a frozen, pretrained image tower with only the text tower trained from scratch — and SigLIP applies it to standard from-scratch (or unlocked fine-tuning) contrastive pre-training of both towers, following the CLIP setup. Both use a ViT image encoder and a Transformer text encoder.

Fig. 2: sigmoid loss (green) beats softmax (orange) by a wide margin below 16k batch size and matches or slightly exceeds it beyond that. All three setups (SigLiT, SigLIP, multilingual mSigLIP) saturate around 32k batch size — scaling to one million (SigLiT) or 307k (SigLIP) gives at most a minor boost, and very large batches (e.g. 307k) can even hurt.

Architecture

• Image encoder: Vision Transformer (ViT), various sizes — B/16, B/8, L/16, g/14, So400m/14 (the Shape-Optimized 400M-parameter architecture from “Getting ViT in Shape”), evaluated at multiple input resolutions and patch counts (196–1024 patches).
• Text encoder: standard Transformer, sizes matched to the image tower (Base/Large/So400m-equivalent); a 32k-token English SentencePiece vocabulary trained on C4, sequences capped at 16 text tokens for the main SigLIP experiments (64 for the scaled-up runs). For multilingual mSigLIP, a larger 250k-token multilingual vocabulary is used with a bottlenecked token embedding ($N\times K$ embedding matrix plus a $K\times W$ projection, $K\ll W$) to keep the embedding table small — $K=96$ costs only ~0.5% ImageNet zero-shot accuracy versus the full un-bottlenecked 250k vocabulary.
• Embeddings from both towers are L2-normalized before the dot product, scaled by the learned temperature $t$, and offset by the learned bias $b$.
• Optimizer: ScalingViT-Adafactor by default; the LION optimizer is used for the small-chip SigLiT/SigLIP experiments (Sections 4.4–4.5) for its memory efficiency.

Hyperparameters

Transcribed from Sections 4.1–4.6 and Table 1/3.

SigLiT (frozen image tower, four TPU-v4 chips)

Image tower	frozen public ⚛B/8 checkpoint, or g/14
Text tower	B-sized (12-layer variant of L), from scratch
Batch size	32k (B/8) or 20k (g/14)
Training time	1 day (B/8) or 2 days (g/14)
Result	79.7% (B/8) / 84.5% (g/14) 0-shot ImageNet

SigLIP from-scratch, small-chip regime (Section 4.4)

Chips	4 TPU-v4
Image tower init	public unlocked ⚛B/16 (ViT-AugReg-B/16), precomputed embeddings
Text tower	Large Transformer but only 12 layers deep (instead of 24)
Optimizer	LION, decoupled weight decay $1\times 10^{-7}$
LR schedule	linear warm-up to peak $1\times 10^{-4}$ over 6.5k steps, then cosine decay to 0
Total steps	65,000
Batch size	32k
Training time	just under 1 day
Result	79.7% 0-shot ImageNet

Key fine-tuning recipe fix: fine-tuning a locked/pretrained image tower degrades its representation quality unless weight decay is disabled specifically on the pre-trained image-tower weights (weight decay is kept only on the randomly-initialized text tower). This single change is what makes unlocked fine-tuning competitive; a 0.1 learning-rate multiplier is also applied to the pre-trained tower.

SigLIP main experiments (Section 4.2/4.6)

Dataset	WebLI (English image-text pairs)
Image resolution	$224\times 224$ (main), $256\times 256$ (scaled-up, $(256/16{)}^2$ patches)
Text tokenizer	32k-vocab SentencePiece (English C4)
Max text tokens	16 (main), 64 (scaled-up)
Batch size	32k (chosen as the near-optimal operating point across all sweeps)
Bias / temperature init	$b=-10$, $t^{\prime }=\log 10$ (i.e. $t=10$)
Adam / AdaFactor ${\beta }_2$	0.95 (reduced from the default 0.999 to stabilize large-batch training)
Scaled-up training length	40 billion examples seen
High-res fine-tune	5 billion further examples, learning rate reduced $100\times$, no weight decay

mSigLIP (multilingual, Section 4.3/4.6)

Dataset	WebLI, all 100 languages
Vocabulary	250k-token multilingual SentencePiece, bottleneck $K=96$, $W=768$
Examples seen	900M (vocabulary ablation) / 30B (main sweep) / scaled-up further in §4.6
Batch size	32k found sufficient; larger batches (up to 245k) do not help and can hurt XM3600 retrieval

Results

Zero-shot ImageNet classification and MSCOCO retrieval, comparing publicly released models at matched configurations (Table 3):

Method	ViT size	Patches	IN-1k val	IN-v2	IN-ReaL	ObjectNet	COCO I→T	COCO T→I
CLIP	B	196	68.3	61.9	–	55.3	52.4	33.1
OpenCLIP	B	196	70.2	62.3	–	56.0	59.4	42.3
EVA-CLIP	B	196	74.7	67.0	–	62.3	58.7	42.2
SigLIP	B	196	76.2	69.6	82.8	70.7	64.4	47.2
SigLIP	B	1024	79.2	73.0	84.9	74.7	67.6	50.4
CLIP	L	576	76.6	72.0	–	70.9	57.9	37.1
EVA-CLIP	L	576	80.4	73.8	–	78.4	64.1	47.9
SigLIP	L	576	82.1	75.9	87.0	81.0	70.6	52.7
EVA-CLIP	E (5B params)	729	82.0	75.7	87.5	79.6	68.8	51.1
SigLIP	So (400M params)	729	83.2	77.2	87.5	82.9	70.2	52.0

SigLIP-L beats all listed models including the much larger EVA-CLIP-E (5B parameters) and CLIPA-v2-H (630M), and the Shape-Optimized SigLIP-So400m matches or exceeds models several times its size — the headline claim that the sigmoid loss’s efficiency gains translate into a better accuracy/compute trade-off, not just cheaper training. The scaled-up multilingual mSigLIP ViT-B sets a new state of the art on XM3600 text→image retrieval at 34.9% recall@1, over 6 points above the prior LiT-based result (28.5%) that used a much larger 4B-parameter ViT-e.

Fig. 6: keeping only the hardest negatives (orange) tracks the unmasked baseline (1:16k) almost exactly, while keeping only the easiest (green) collapses catastrophically — most of the useful learning signal comes from hard negatives, suggesting an efficient hard-negative-mining scheme is a promising (if nontrivial) direction.

Key ablations

• Batch size. Sigmoid loss dominates softmax below ~16k batch size, and the two are roughly on par above that; both saturate around 32k, and larger batches (up to the extreme of 1M for SigLiT) buy only minor further gains — a very large batch (e.g. 307k for SigLIP) can actively hurt.
• ${\beta }_2$ and stability. As batch size grows, training becomes increasingly unstable, seen as periodic spikes in the loss and gradient norm; reducing Adam/AdaFactor’s ${\beta }_2$ from 0.999 to 0.95 recovers from spikes faster and stabilizes large-batch training without removing the spikes entirely.
• Bias term. Disabling $b$, or badly initializing it (0 instead of $-10$), consistently hurts, especially when paired with a small $t^{\prime }$ initialization — the bias prevents large early over-correction from the positive/negative imbalance.
• Negative ratio / batch composition. Masking down to a near-balanced 1:1.6 positive:negative ratio by keeping only the hardest negatives preserves quality almost exactly; keeping only the easiest negatives collapses training. The imbalance itself is not the core issue — most useful gradient signal comes from hard negatives, and an efficient hard-negative-mining scheme is flagged as promising future work.
• Label / data-noise robustness. Under synthetic corruption of images, text, batch alignment, or combinations thereof, sigmoid-trained models retain a consistent accuracy edge over softmax-trained models as corruption probability increases (Fig. 7) — a property previously observed for sigmoid losses in supervised classification, now shown to transfer to contrastive image-text pre-training.

Fig. 7: sigmoid loss (green) stays above softmax (orange) across every corruption type and severity tested — noise robustness that plausibly matters given how noisy web-scraped image-text pairs are in practice.

Limitations

• Gains over softmax are concentrated at small-to-moderate batch sizes; at the large batch sizes historically used by the biggest contrastive pre-training runs (BASIC, LAION-style ≥64k), softmax quickly catches up and the two losses perform similarly.
• Very large batch sizes (100k+) can hurt both losses, and scaling beyond ~32k gives diminishing or even negative returns, undercutting the “just add more compute/larger batch” scaling intuition common in this literature.
• The paper deliberately does not propose an efficient hard-negative mining scheme, despite showing hard negatives carry most of the useful signal — this is left as future work.
• Fine-tuning a locked/pretrained image tower still requires the ad hoc fix of disabling weight decay specifically on those weights; the underlying reason fine-tuning otherwise degrades representation quality is not fully explained, only empirically worked around.
• Evaluation is zero-shot classification/retrieval and multilingual retrieval; the paper does not report dense-prediction transfer (detection/segmentation) of the resulting encoders, unlike some contemporaneous CLIP-family studies.

See also

• CLIP — the softmax-based contrastive baseline SigLIP directly compares against and inherits its architecture and naming conventions from
• SigLIP2 — follow-up work that builds directly on the sigmoid loss and SigLIP training recipe
• PaliGemma — a vision-language model that uses a SigLIP-trained vision encoder (So400m) as its image tower
• How to train your ViT — the ViT training/regularization recipe underlying the image towers used here
• Image Captioners Are Scalable Vision Learners Too — a generative alternative to contrastive image-text pre-training, evaluated in a similar setting
• TinyViT — a different angle on efficient ViT training, for small backbones rather than efficient contrastive losses