PaliGemma

PaliGemma is an open 3B-parameter vision-language model (VLM) built by combining a SigLIP image encoder with the Gemma-2B language model.

• Paper: Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, and a large team at Google DeepMind, PaliGemma: A versatile 3B VLM for transfer, arXiv:2407.07726, July 2024.
• Lineage: the fourth entry in the PaLI line of vision-language models (PaLI, PaLI-X, PaLM-E, PaLI-3).
• Goal: not to be the best model out of the box, but the best base model for transfer; released without instruction-tuning, it is meant to be fine-tuned onto a target task.
• Headline result: demonstrated on more than 30 academic benchmarks (captioning, VQA, remote sensing, video QA, counting, and referring-expression segmentation), often matching or beating the much larger PaLI-X (55B) and PaLM-E (562B).

Fig. 1: SigLIP encodes the image into soft tokens, a linear layer projects them into Gemma’s embedding space, and Gemma-2B autoregressively decodes the answer.

Motivation: a small, versatile, transferable VLM

PaliGemma’s design philosophy is the opposite of scaling up: PaLI-X and PaLM-E chased performance by growing the vision and language components to 22B and 540B parameters respectively, but PaLI-3 showed that a well-trained small 5B model (2B vision + 3B language) can match those giants across most vision-language tasks. PaliGemma pushes this further, combining a 400M SigLIP encoder with a 2B Gemma decoder for well under 3B total parameters, and asks a narrower question than “how good is this model zero-shot”: how good a starting point is it once fine-tuned? The paper is explicit that none of the ~40 evaluation tasks or their images are part of pretraining, and that the model is deliberately released without instruction-tuning or chat behavior, “to serve as a useful starting point for further research… and encourage a clearer separation of base models and fine-tunes in VLM research.”

Architecture

PaliGemma consists of three components (Fig. 1):

• Image encoder: the “shape-optimized” SigLIP ViT-So400m checkpoint, contrastively pretrained at scale with the sigmoid loss. PaliGemma uses this rather than a larger model like ViT-G (used by PaLI-X) specifically because of SigLIP’s strong performance-per-parameter — see SigLIP for the sigmoid-loss pretraining objective and shape-optimization procedure that produces this specific checkpoint, and How to train your ViT for the broader ViT-training design space it draws on.
• Language model: the public Gemma-2B v1.0 raw pretrained (not instruction-tuned) checkpoint, a decoder-only transformer from the same lineage as Gemini. Gemma-2B is chosen over the 7B/70B variants as “a great balance between size and performance” — larger LMs help mainly on tasks like mathematical reasoning that are not the paper’s focus.
• Connector: a single linear layer projecting SigLIP’s output token embeddings into Gemma’s vocabulary embedding dimension, initialized to zero. An ablation (Section 5.5) compares this against an MLP connector with a GeLU hidden layer and finds them statistically indistinguishable (77.2 vs 77.1 average transfer score when both are fully fine-tuned), so the paper keeps the simplest option.

The image is resized to a fixed square resolution — 224, 448, or 896 px, depending on checkpoint — and produces a fixed number of image tokens ($N_{\text{img}}=$ 256, 1024, or 4096 respectively). Text is tokenized with Gemma’s SentencePiece tokenizer. The input sequence to the decoder is constructed as

tokens = [image tokens..., BOS, prefix tokens..., SEP, suffix tokens..., EOS, PAD...]

where the prefix is the task/query text (e.g. a question) and the suffix is the target text (the answer) that the model is trained to generate autoregressively. \n is used as the SEP token and is tokenized separately so it cannot be merged with adjacent prefix/suffix tokens by the tokenizer.

Prefix-LM attention

Fig. 2: image and prefix tokens attend to each other with full, unmasked (bidirectional) attention; suffix tokens are generated with a standard causal mask and can additionally attend back to the full image+prefix block.

Most decoder-only VLMs simply apply a causal mask over the whole concatenated sequence. PaliGemma instead uses a prefix-LM scheme: image and prefix tokens attend to each other with full, bidirectional (block) attention, while suffix tokens are covered by the usual causal mask and can additionally look back at the entire image+prefix block. Concretely, writing the attention mask $M(i,j)\in \{0,-\infty \}$ added to the logits before the softmax,

$$M(i,j)=\{\begin{array}{ll}0& \text{if }j\le T_{\text{pre}}\text{ (image+prefix tokens can attend anywhere within the prefix block)}\\ 0& \text{if }i>T_{\text{pre}}\text{ and }j\le i\text{ (suffix tokens are causal, and can see the full prefix)}\\ -\infty & \text{otherwise}\end{array}$$

where $T_{\text{pre}}$ is the combined length of image and prefix tokens. The motivation is that bidirectional attention lets image tokens “look ahead” at the query/prefix and update their representation for the task at hand from the very first layer, rather than only ever attending to earlier image patches. Ablation (Fig. 5, Section 5.2) confirms this empirically: restricting attention to autoregressive-only on the prefix, or extending the autoregressive mask onto the image tokens too, both perform clearly worse than the prefix-LM setting. The next-token loss is applied only on the suffix — applying it to the prefix as well (in effect asking the model to also “guess the question”) also measurably hurts average performance.

Detection and segmentation as text

Rather than special-casing dense-prediction heads, PaliGemma follows PaLI-3/LocCa and pix2seq in converting bounding boxes and segmentation masks into text, via new vocabulary tokens added on top of Gemma’s SentencePiece vocabulary:

• 1024 location tokens <loc0000>–<loc1023>, each corresponding to a normalized image coordinate binned into 1024 bins. A detection box is written as four such tokens <ymin><xmin><ymax><xmax>, following the same coordinate convention used in the DETR family’s normalized-box outputs conceptually, though here the boxes are literal subword tokens the decoder emits rather than regression targets. Multi-object detection is expressed as detect {thing} ; {thing} ; ..., close to Pix2Seq’s sequence formulation.
• 128 mask tokens <seg000>–<seg127>, a VQ-VAE-tokenized codebook for a single object’s mask (following PaLI-3 and the codebook used for SAM-style mask tokenization), so referring-expression segmentation is expressed as a location box (4 loc tokens) immediately followed by mask tokens, e.g. for the prompt “the giraffe on the right standing tall” the target suffix is literally <loc0347><loc0553><loc0788><loc0749><seg093><seg106>....

New-token initialization matters: standard small-Gaussian init ($\sigma =0.02$) gives worse initial loss on tasks using these tokens than initializing them near the average of existing embeddings (AvgEmb, as proposed in prior work), but that gap closes within a few thousand steps, and $\sigma =0.02$ ends up with both better Stage-1 perplexity and better downstream RefCOCO-segmentation mIoU after full training — so PaliGemma keeps plain Gaussian init (Fig. 6).

This “detection/segmentation-as-text” approach means the same generic image+text-in, text-out decoding API used for captioning and VQA also covers structured spatial outputs, at the cost of relying on the small quantization granularity of 1024 location bins and a 128-entry mask codebook rather than continuous regression, unlike DETR-style set-prediction detectors or dedicated segmentation architectures.

Training recipe: four stages

PaliGemma follows the same multi-stage recipe as previous PaLI models:

• Stage 0 — unimodal pretraining. No custom unimodal pretraining is done for PaliGemma itself; it simply reuses the existing, independently-pretrained SigLIP-So400m and Gemma-2B checkpoints, so as to benefit from their own well-studied, separately-scaled training recipes.
• Stage 1 — multimodal pretraining. The combined model (nothing frozen — see below) is trained at 224px resolution ($N_{\text{img}}=256$) with text sequence length $N_{\text{txt}}=128$, on a broad mixture of large-scale vision-language tasks (captioning, OCR, VQA/VQG, open-vocabulary detection and segmentation on pseudo-labeled data), for 1 billion examples total. A slow linear warm-up on the image encoder’s learning rate (Fig. 3) protects SigLIP’s pretrained quality from the initially-misaligned gradients coming through the still-untrained connector and LLM.
• Stage 2 — resolution increase. Two further short continued-pretraining runs “upcycle” the Stage-1 checkpoint to 448px (+50M examples) and then 896px (+10M examples), using the same task mixture as Stage 1 but with resolution-sensitive tasks (OCR, detection, segmentation) upweighted and, for tasks like OCR, much longer suffix sequences ($N_{\text{txt}}=512$).
• Stage 3 — transfer. The three resulting base checkpoints (224px/448px/896px) are fine-tuned end-to-end (all parameters unfrozen) on each individual downstream task with a simple, small transfer recipe.

Unlike the common practice of freezing the image encoder during multimodal pretraining (to avoid degrading its contrastively-learned representation, as noted in LiT), PaliGemma leaves everything trainable at every stage, following evidence from CapPa and LocCa (see Image Captioners Are Scalable Vision Learners Too) that captioning-style objectives can teach a vision encoder spatial/relational understanding that contrastive pretraining alone misses. An ablation over freezing combinations (Fig. 7) confirms this is the best choice: keeping the ViT frozen makes no difference after transfer, but degrades pretraining perplexity on spatially-demanding tasks; freezing the LLM is clearly worse; and resetting any component before training is much worse still, confirming that starting from the pretrained Stage-0 checkpoints (rather than training from scratch) is important.

An “infinite” rsqrt learning-rate schedule (no decay between stages, Fig. 3) chains Stage 0 through Stage 2 as one continuous pretraining curve; transfer (Stage 3) then acts as the cooldown, annealing the learning rate to zero with a short linear warm-up and cosine decay.

Pretraining task mixture

Stage 1/2 pretraining uses a fixed set of task prefixes (each task gets a unique prefix string to avoid conflicting learning signals across skills):

Prefix	Task
caption {lang}	Plain captioning (WebLI in 100+ languages, CC3M-35L)
ocr	Concatenation of all OCR-transcribed text in raster order
answer en {question}	English VQA (generated from CC3M-35L captions) plus English object-centric questions on OpenImages (presence, listing, counting, multi-object presence)
question {lang} {English answer}	Generated VQG in 35 languages
detect {thing} ; {thing} ; ...	Multi-object detection on pseudo-labeled open-world data (via OWL-ViTv2)
segment {thing} ; {thing} ; ...	Multi-object instance segmentation (via OWL-ViTv2 + SAM pseudo-labels)
caption <ymin><xmin><ymax><xmax>	Grounded captioning of the boxed region (LocCa-style)

Notably, none of PaliGemma’s pretraining data is distilled from a larger commercial VLM (unlike, e.g., LLaVA’s GPT-4-generated instruction data). All near-duplicate images of the transfer-benchmark evaluation sets, and of several popular CV benchmarks generally, are explicitly removed from the web-scale pretraining corpus.

Hyperparameters

Stage 1 (multimodal pretraining)

Resolution	224px ($N_{\text{img}}=256$)
Text sequence length	$N_{\text{txt}}=128$
Examples seen	1 billion
LR schedule	continuous rsqrt (“infinite”), slow linear warm-up specifically for the image encoder
Frozen components	none

Stage 2 (resolution increase)

Checkpoint	Additional examples	Notes
448px	+50 M	same task mixture as Stage 1, resolution-sensitive tasks upsampled
896px	+10 M	same, plus longer suffix length ($N_{\text{txt}}=512$) for tasks like OCR

Stage 3 (transfer) — recommended sweep ranges, bold = recommended initial value

Hyperparameter	Values explored
Resolution (checkpoint)	224, 448, 896
Epochs	1, 3, 10, 30, 100
Learning rate	3e-5, 1e-5, 3e-6
Label smoothing	0.0, 0.1, 0.3
Dropout in the LLM	0.0, 0.1, 0.3
Weight decay	0.0 or 0.1 × learning rate
Freeze ViT	false, true
Beam search	may benefit captioning

The paper additionally shows (Section 6.2, Table 2) that a single fixed simple recipe — lr = 1e-5, batch size 256, no dropout, no label smoothing, no weight decay, nothing frozen, only the number of epochs (best-of, capped at 10) tuned per task — already reaches within 2.5% relative regret of the fully-tuned result on 37 of the ~40 tasks; only RefCOCO(+/g) and SciCap benefit clearly from more epochs, label smoothing, and dropout. A handful of representative per-task Stage-3 configurations from Appendix J illustrate the range actually used:

Task	Res	Epochs	Batch	LR	Label smoothing	Freeze ViT	Decode
COCOcap	224	5	256	1e-5	0.0	False	beam n=2
VQAv2	224	10	256	1e-5	0.0	False	greedy
ChartQA	224	30	256	1e-5	0.2	False	greedy
RefCOCO (seg.)	224	100	256	3e-5	0.3	False	greedy
DocVQA	896	10	256	1e-5	0.0	False	greedy
ScienceQA	224	20	128	1e-5	0.0	True	greedy

One training run of the final model on a TPUv5e-256 pod takes just under 3 days for Stage 1 and 15h for each Stage 2 step; individual task transfers take between 20 minutes and 10 hours on a TPUv3-32.

Results

Table 1 (headline numbers, best single of 5 seeds; ⌐ marks zero-shot evaluation of a transferred checkpoint on a different benchmark):

Task group	Representative results (224px → 448px → 896px)
Captioning	COCOcap 141.9 → 144.6; NoCaps (0-shot) 121.7 → 123.6; TextCaps 127.5 → 153.9
VQA	VQAv2 83.2 → 85.6; ScienceQA 95.4 → 95.9; ChartQA (human) 40.0 → 54.2; DocVQA 43.7 → 78.0 → 84.8; InfoVQA 28.5 → 40.5 → 47.8
Segmentation	RefCOCO (testA) 75.7 → 77.9 → 78.7 mIoU; RefCOCOg (test) 68.2 → 71.0 → 72.7 mIoU
Video (224px only)	ActivityNet-QA 50.8; MSRVTT-CAP 70.5; VATEX 79.7

Fig. 3: full Table 1, split by task family and resolution checkpoint. Highlighted rows are the tasks the paper identifies as resolution-sensitive — text-heavy document/chart/infographic understanding gains the most from 448px → 896px.

Document- and text-heavy tasks are the clearest beneficiaries of higher resolution (DocVQA nearly doubles, 43.7 → 84.8, from 224px to 896px), while most captioning/VQA/video tasks are comparatively resolution-insensitive. An ablation (Fig. 9) disentangles why resolution helps: for resolution-sensitive tasks roughly half the gain comes from the extra information content of the higher-res image, and half from the extra model capacity of the longer resulting token sequence.

Repeatability. Across 5 re-runs with different seeds, the standard deviation of most tasks’ final metric is below 0.5 points (Fig. 11), and this holds both for Stage-1 pretraining variance and for transfer variance — with the notable exception of RefCOCO-family segmentation, whose mIoU can vary 10–30% across seeds/example ordering at limited data.

Data efficiency. Fig. 12 shows the relative regret of fine-tuning on only 64/256/1024/4096 examples versus the full training set (30k–1M examples, depending on task): with as few as 4,096 examples the median task is within 10% of the full-data score, and with 256 examples within about 20% — i.e. PaliGemma can often be prototyped on a new task with a genuinely small labeled set.

Fig. 4: each light line is one task; the thick black line is the median across all ~40 tasks. Diminishing but still substantial data-efficiency gains persist out to 4,096 examples.

Notable individual results. PaliGemma at 224px reaches 47.3% paired accuracy on MMVP (a benchmark of CLIP-blind image pairs), state-of-the-art “by a large margin” versus GPT-4V (38.7%) and Gemini (40.7%) at the time, and other tested models including LLaVA score below chance. The paper also reports (Appendix G) surprising zero-shot generalization to 3D renders from Objaverse despite never being trained on synthetic 3D data.

Ablations of note

• Connector: linear vs. MLP is a wash (Section 5.5, above).
• Vision encoder or none at all: removing SigLIP entirely and feeding a linear projection of raw RGB patches directly into Gemma (Fuyu-style) still works, but is dramatically less sample-efficient — it needs far more pretraining examples to approach (and still trails) the SigLIP-encoder model’s average score across 37 tasks (Fig. 8), which the authors attribute partly to SigLIP already having seen 40B image-text pairs during its own Stage-0 pretraining.
• Resolution vs. windowing: rather than a single high-resolution checkpoint with a “windowed attention” trick to reduce cost (splitting a 448px image into four 224px crops encoded separately), PaliGemma’s approach of dedicated Stage-2 checkpoints per resolution performs clearly better, at the cost of shipping three separate checkpoints.
• Plain resize beats aspect-preserving crop augmentation for segmentation: simple resize-to-square (224×224) works as well as more elaborate zoom/crop augmentations for RefCOCO segmentation (Appendix C).
• Image annotation vs. text prompt: marking the referred widget with a literal red box drawn on the image gives results comparable to indicating the same region with <loc> tokens in the text prompt (Appendix F).

Limitations

• PaliGemma is explicitly not instruction-tuned or chat-aligned — it is a base model whose intended use is task-specific fine-tuning, not zero-shot conversational use (the strong zero-shot numbers reported, e.g. MMVP, NoCaps, are evaluated on checkpoints already transferred to a related task, not the untouched pretrained base).
• Fixed-resolution, fixed-square image handling (no aspect-ratio preservation, no flexible patch count as in FlexiViT/NaViT) means a separate checkpoint per resolution is needed, and the paper explicitly confirms a single checkpoint adapted at inference-time to another resolution performs clearly worse than a natively-trained one.
• Location and segmentation are quantized to 1024 coordinate bins and a 128-entry mask-token codebook respectively — coarser than continuous-regression detection/segmentation heads, trading structured-output flexibility for reuse of the same text-decoding interface.
• Some pretraining datasets used (e.g. internal WebLI subsets) remain private / not fully released, and the paper notes RefCOCO-style segmentation transfer has materially higher run-to-run variance than other tasks even with identical hyperparameters.
• Larger LLMs (7B/70B) are explicitly not used, and the paper notes they would likely help specifically on tasks requiring stronger reasoning (e.g. math), which is outside PaliGemma’s target task mix.

See also

• SigLIP — the vision encoder PaliGemma uses (ViT-So400m, sigmoid contrastive loss); PaliGemma keeps it fully unfrozen through all pretraining stages rather than freezing it as prior VLMs do
• SigLIP2 — later evolution of the SigLIP recipe
• How to train your ViT — the broader ViT pretraining/regularization design space SigLIP draws on
• Image Captioners Are Scalable Vision Learners Too — the CapPa/captioning-objective evidence that motivates not freezing the vision encoder during multimodal pretraining
• DETR — set-prediction detection, contrasted with PaliGemma’s detection-as-text-tokens formulation
• SAM / SAM2 / SAM3 / MobileSAM / PerSAM — dedicated segmentation models, contrasted with PaliGemma’s mask-as-VQ-VAE-tokens approach to referring-expression segmentation
• TinyViT — a small vision backbone built by a very different (distillation-based) route to efficiency