Your ViT is Secretly an Image Segmentation Model

EoMT (Encoder-only Mask Transformer) is a panoptic/semantic/instance segmentation model that uses a plain ViT with no task-specific machinery.

• Paper: Kerssies, Cavagnero, Hermans, Norouzi, Averta, Leibe, Dubbelman and de Geus (TU Eindhoven / Polytechnic of Turin / RWTH Aachen), Your ViT is Secretly an Image Segmentation Model, CVPR 2025, arXiv:2503.19108, code at tue-mps.org/eomt.
• Claim (negative and deflationary): the task-specific machinery that state-of-the-art ViT segmentation models bolt onto a plain ViT encodes inductive biases that a sufficiently large, sufficiently pre-trained ViT has already learned for itself.
• Machinery removed: a convolutional adapter (ViT-Adapter) for multi-scale features, a pixel decoder to fuse those scales, and a Transformer decoder with masked cross-attention (Mask2Former, M2F).
• Core idea: add $K$ learnable query tokens and a small mask head to a plain ViT; accuracy barely moves while speed multiplies.
• Headline result: EoMT with ViT-L reaches 56.0 PQ at 128 FPS on COCO val2017, against ViT-Adapter-L + Mask2Former’s 57.1 PQ at 29 FPS, i.e. $4.4\times$ faster for $1.1$ PQ.

Left: what the two pipelines contain. Right: Panoptic Quality vs. FPS (log scale) across DINOv2-pre-trained ViT sizes on COCO val2017. EoMT occupies the whole right side of the curve.

Motivation

The standard recipe for ViT segmentation is a stack of compensations for the ViT’s two “deficiencies” relative to a CNN: a ViT is single-scale (one patch size, one resolution) and globally-attentive (no local convolutional prior). ViT-Adapter injects convolutional multi-scale features into the ViT via multi-scale deformable attention (see Deformable DETR) and extracts $\{{\mathbf{F}}_4,{\mathbf{F}}_8,{\mathbf{F}}_{16},{\mathbf{F}}_{32}\}$; Mask2Former’s pixel decoder fuses them into $\{{\hat{\mathbf{F}}}_4,{\hat{\mathbf{F}}}_8,{\hat{\mathbf{F}}}_{16},{\hat{\mathbf{F}}}_{32}\}$; the Transformer decoder runs $J$ blocks of masked cross-attention plus query self-attention over $K$ learnable queries.

The paper’s two hypotheses:

1. Large-scale pre-training already provides the fine-grained information. Masked-image-modeling objectives such as DINOv2 and EVA-02 teach the ViT to extract dense, semantically fine-grained patch features — exactly what these components were added to supply.
2. Larger ViTs have the capacity to do the rest themselves. With more learnable parameters, the ViT can absorb the segmentation-specific computation that the decoder was doing.

If both hold, the components are redundant, and their cost — which is disproportionately in latency, not FLOPs — is pure waste.

Preliminaries

A ViT splits $\mathbf{I}\in {\mathbb{R}}^{3\times H\times W}$ into $N$ non-overlapping $p\times p$ patches, linearly projects them into patch tokens ${\mathbf{X}}^0\in {\mathbb{R}}^{D\times N}$, and applies $L$ blocks

${\mathbf{Z}}^i={\mathbf{X}}^i+\text{MHSA}(\text{Norm}({\mathbf{X}}^i)),{\mathbf{X}}^{i+1}={\mathbf{Z}}^i+\text{MLP}(\text{Norm}({\mathbf{Z}}^i))$

with Norm layer normalization. Reordering the final tokens spatially yields ${\mathbf{F}}^{\text{vit}}\in {\mathbb{R}}^{D\times \frac{H}{p}\times \frac{W}{p}}$.

In a Mask Transformer, queries ${\mathcal{Q}}^J$ produce per-query class logits ${\mathbf{c}}_i^J\in {\mathbb{R}}^C$ from a linear layer and mask logits ${\mathbf{m}}_i^J\in {\mathbb{R}}^{\frac{H}{4}\times \frac{W}{4}}$ by an MLP yielding a mask embedding ${\hat{\mathbf{q}}}_i^J$ and a dot product with the features ${\hat{\mathbf{F}}}_4$. Each ground-truth segment is assigned to a unique query.

Removing the task-specific components

Fig. A of the paper: configurations (0)–(3) of the ablation ladder, before the decoder itself is dissolved into the ViT.

The paper strips the pipeline one stage at a time (Tab. 1, ViT-L, DINOv2, COCO val2017):

• (1) Remove the adapter. Build the feature pyramid directly from ${\mathbf{F}}^{\text{vit}}={\mathbf{F}}_{16}$, following ViTDet: for each scale, repeat a (transposed) $2\times 2$ convolution with stride $2\times 2$, GELU, a depthwise $3\times 3$ convolution, and a final Norm until the target resolution is reached — upscaling to ${\mathbf{F}}_4,{\mathbf{F}}_8$, downscaling to ${\mathbf{F}}_{32}$.
• (2) Remove the pixel decoder. With all scales derived from one backbone map, there is nothing left to consolidate across hierarchy stages; feed $\{{\mathbf{F}}_4,{\mathbf{F}}_8,{\mathbf{F}}_{16},{\mathbf{F}}_{32}\}$ straight to the Transformer decoder.
• (3) Remove multi-scale processing. Queries cross-attend exclusively to ${\mathbf{F}}^{\text{vit}}={\mathbf{F}}_{16}$; only ${\mathbf{F}}_4$ is still computed, purely to keep mask logits at high resolution.

#	Configuration	Params	GFLOPs	FPS	PQ
(0)	ViT-Adapter + Mask2Former	349M	830	29	57.1
(1)	w/o ViT-Adapter	342M	700	36	56.7
(2)	w/o pixel decoder	337M	685	61	56.9
(3)	w/o multi-scale	328M	673	64	56.7
(4)	w/o Transformer decoder	316M	828	61	56.2
(5)	w/o masking = EoMT	316M	669	128	56.0

Steps (1)–(3) cost $0.4$ PQ and buy a $2.2\times$ speedup. The FPS gain far outruns the FLOPs gain — the removed components are latency-heavy relative to their arithmetic, because they break the highly optimized plain-ViT execution path (FlashAttention, torch.compile, fused kernels) with custom ops. Step (4) increases FLOPs and lowers FPS, since folding the decoder into the ViT means the mask module is applied repeatedly for masked attention, with upscaling dominating; step (5) removes masked attention at inference and recovers everything.

Architecture

Learnable queries are concatenated to the patch tokens after $L_1$ ViT blocks; the last $L_2$ blocks process both jointly and feed the mask module.

EoMT is a plain ViT plus two small additions:

• Queries. After the first $L_1$ unchanged encoder blocks, $K$ learnable query tokens are concatenated to the patch token sequence. The remaining $L_2$ blocks process the concatenated sequence with ordinary MHSA per Eq. (1). A standard Mask Transformer separates (i) query–query self-attention and (ii) query-to-patch cross-attention into sequential operations; EoMT’s single self-attention over the joint sequence performs both at once, at zero architectural cost. The last blocks therefore do double duty: process patch tokens as before and replace the Transformer decoder.
• Mask module. As in Mask2Former: a linear layer on the query tokens for class logits, and a three-layer MLP followed by a dot product with the upscaled features ${\mathbf{F}}_4$ for mask logits.

The token overhead is small: adding 200 queries to a model processing $640\times 640$ images at patch size $16\times 16$ is $+12.5\%$ tokens per block, but only for the last $L_2$ blocks. With $L_1=20$, $L_2=4$ for ViT-L, total tokens processed across the ViT rise by only $2.1\%$.

Masked attention and mask annealing

The joint self-attention matrix over queries and patch tokens. Only the query-to-patch quadrant is masked, mimicking M2F’s masked cross-attention.

Mask2Former’s masked cross-attention (a query attends only to its own predicted mask region) improves accuracy but requires predicting and applying an intermediate mask before every decoder block. EoMT can reproduce it during training: the shared mask module is applied before each of the last $L_2$ blocks, and the predicted mask gates the query-to-patch quadrant of the joint self-attention matrix. Query-to-query, patch-to-query and patch-to-patch attention stay unmasked.

Doing this at inference is exactly the expense that ruins step (4) above. The fix is mask annealing: for each block, apply the mask for a given query with probability $P_{\text{mask}}$, starting at $1.0$ and decaying block by block to $0.0$ over training, earlier blocks annealing first. The model benefits from masked attention’s faster convergence early on, then learns to operate without it, and at inference is a plain ViT with no intermediate modules and no modified attention op.

Masking probability per block over 12 epochs (ViT-L, $L_2=4$, blocks 21–24). Annealing is staggered: earlier blocks are freed first.

Ablating the alternatives (Tab. 7 / Tab. C, COCO val2017):

Training	Inference	GFLOPs	FPS	PQ
Masking	Masking	828	61	56.2
Masking	w/o masking	669	128	27.4 ($-28.8$)
w/o masking	w/o masking	669	128	53.2 ($-3.0$)
Mask annealing	w/o masking	669	128	56.0 ($-0.2$)

Simply disabling masking at inference after training with it is catastrophic ($27.4$ PQ) — a train/test mismatch, not a free lunch. Never masking is merely bad ($53.2$). Annealing gets essentially the masked-training PQ at masked-free inference speed, more than doubling throughput. The same trick also works for the ViT-Adapter + M2F baseline (Tab. C: $56.8$ vs $57.1$).

Hyperparameters

Section 4.1 and Appendix A.

Setting	Value
Default backbone	DINOv2-L (vit_large_patch14_reg4_dinov2 via timm), $640\times 640$ input, patch size $16\times 16$
Other backbones	DINOv2 ViT-g/L/B/S, EVA-02-L, DeiT-III ViT-L (IN21K and IN1K)
$L_2$ (blocks processing queries)	3 for ViT-S/B, 4 for ViT-L, 5 for ViT-g
Queries $K$	200 for panoptic and instance, 100 for semantic
Optimizer	AdamW, learning rate $10^{-4}$, mixed precision
Layer-wise LR decay (LLRD)	factor 0.8
LR decay	polynomial, factor 0.9
Mask annealing	polynomial, factor 0.9; $P_{\text{mask}}$ from 1.0 to 0.0
LR warm-up	two-stage linear: 500 iterations warming up the randomly initialized parameters with pre-trained weights frozen, then 1000 iterations warming up the pre-trained parameters; initial LR 0 in both cases
Epochs	12 ($1\times$ schedule) on COCO, 31 on ADE20K, 107 on Cityscapes
Loss	${\mathcal{L}}_{\text{tot}}={\lambda }_{\text{bce}}{\mathcal{L}}_{\text{bce}}+{\lambda }_{\text{dice}}{\mathcal{L}}_{\text{dice}}+{\lambda }_{\text{ce}}{\mathcal{L}}_{\text{ce}}$ with ${\lambda }_{\text{bce}}=5.0$, ${\lambda }_{\text{dice}}=5.0$, ${\lambda }_{\text{ce}}=2.0$ (as Mask2Former)
Augmentation	Mask2Former’s: horizontal flip, scale jitter, padding, random crop; color jitter additionally on ADE20K and Cityscapes; large-scale jitter ($0.1\times$–$2.0\times$) for panoptic/instance, normal-scale jitter ($0.5\times$–$2.0\times$) for semantic
Patch embedding / pos. emb. resizing	FlexiViT-style kernel resize to $16\times 16$ plus bicubic interpolation of positional embeddings, done before fine-tuning and kept fixed
Inference	padded inference (resize longer side, zero-pad to square) for panoptic/instance; windowed sliding-window inference for semantic
Benchmarking hardware	NVIDIA H100, FlashAttention-2, torch.compile, batch size 1; FLOPs via fvcore

Results

Model size (COCO val2017, Tab. 3)

Size	Method	Params	GFLOPs	FPS	PQ
g	ViT-Adapter + M2F	1209M	2510	20	57.7
g	EoMT w/ masking	1164M	2689	35	57.2
g	EoMT	1164M	2261	55	57.0 ($-0.7$)
L	ViT-Adapter + M2F	349M	830	29	57.1
L	EoMT	316M	669	128	56.0 ($-1.1$)
B	ViT-Adapter + M2F	121M	347	32	54.4
B	EoMT	93M	216	261	50.6 ($-3.8$)
S	ViT-Adapter + M2F	47M	165	33	50.5
S	EoMT	24M	58	330	44.7 ($-5.8$)

This is the paper’s central scaling claim: the gap shrinks monotonically with model size — $5.8$ PQ at ViT-S, $3.8$ at ViT-B, $1.1$ at ViT-L, $0.7$ at ViT-g. Task-specific components matter when the ViT is small; they become nearly irrelevant when it is large. And because EoMT is so much faster, the speed budget that would buy a ViT-B baseline buys a ViT-L EoMT — the trade is favourable in both directions.

Pre-training (COCO val2017, ViT-L, Tab. 2)

Pre-training	ViT-Adapter + M2F PQ	EoMT PQ	Gap
DINOv2	57.1	56.0	$1.1$
EVA-02	56.7	55.5	$1.2$
IN21K (DeiT-III)	53.9	50.0	$3.9$
IN1K (DeiT-III)	50.4	44.3	$6.1$

Same story along the other axis. With supervised ImageNet pre-training the components are worth $4$–$6$ PQ; with masked-image-modeling vision foundation models they are worth ~$1$. The paper attributes this to MIM objectives producing the patch-level semantic understanding segmentation needs.

Benchmarks

• Panoptic (Tab. 4). EoMT ViT-L at $640^2$: 56.0 PQ / 128 FPS on COCO, 50.6 PQ / 128 FPS on ADE20K; at $1280^2$: 58.3 PQ / 30 FPS COCO, 51.7 PQ ADE20K. EoMT ViT-g at $1280^2$: 59.2 PQ / 12 FPS COCO, 52.8 PQ ADE20K. For comparison, Mask2Former + ViT-Adapter-L at $1280^2$ gets 59.7 PQ at 10 FPS (4817 GFLOPs vs EoMT’s 4146), and Mask2Former + Swin-L (IN21K) 57.8 PQ at 24 FPS. On COCO, EoMT is on par with state of the art while up to $2.1\times$ faster.
• Semantic (Tab. 5). Cityscapes val: EoMT ViT-L 84.2 mIoU at 25 FPS (4350 GFLOPs) vs ViT-Adapter-L + M2F 84.5 at 7 FPS (5200 GFLOPs) — $\approx 3.6\times$ faster for $0.3$ mIoU, and up to $4.4\times$ faster elsewhere. ADE20K val: 58.4 mIoU at 92 FPS vs 58.9 at 21 FPS.
• Instance (Tab. 6). COCO val2017: EoMT ViT-L 45.2 AP at 128 FPS ($640^2$) and 48.8 AP at 30 FPS ($1280^2$), against ViT-Adapter-L + M2F 47.6 AP at 29 FPS and 48.9 AP at 10 FPS. The accuracy drop is somewhat larger here than for the other tasks, but the AP/FPS trade-off is still better.

Out-of-distribution generalization

Being ViT-only, EoMT can be initialized from vision foundation models, which are the strongest OOD performers. Trained on Cityscapes train, evaluated on BRAVO (Tab. 8): DINOv2-based models reach $77$–$78$ mIoU${}_{\text{OOD}}$ vs Swin-L + M2F’s $69.4$, despite similar in-distribution mIoU ($\approx 83$–$85$); EoMT gets 84.2 mIoU${}_{\text{ID}}$ / 77.2 mIoU${}_{\text{OOD}}$. On OOD confidence estimation (Tab. E) EoMT reaches AUPRC${}_{\text{OOD}}$ 89.7, far above ViT-Adapter-L + M2F’s 68.7 and Swin-L + M2F’s 56.8 — it assigns low confidence to the whole OOD object rather than to fragments of it.

Token merging (Tab. 9)

Because EoMT stays close to a plain ViT, it inherits generic ViT optimizations. Applying ALGM token merging (no retraining, batch size 32, Cityscapes val) raises throughput from 38 to 50 images/s ($+31\%$) at unchanged 84.2 mIoU. ViT-Adapter + M2F’s FLOPs drop under ALGM (5200 → 3031) but throughput does not move at all ($9$ img/s either way), bottlenecked by its extra components and the cost of “unmerging” tokens for adapter interaction.

$L_2$ ablation (Tab. D)

Number of ViT blocks that also process queries, ViT-L on COCO: $L_2=9\to 55.7$ PQ / 126 FPS; $6\to 55.7$ / 127; $4\to 56.0$ / 128; $2\to 55.4$ / 128. Performance is flat and FPS is essentially unaffected, so $L_2=4$ is chosen as the ViT-L default.

Limitations

• The claim is conditional on scale. At ViT-S and ViT-B, EoMT loses $5.8$ and $3.8$ PQ. The paper’s title is only true for large models; for small ones the adapter and decoder are still doing real work. Similarly, with plain ImageNet pre-training the gap reopens to $6.1$ PQ.
• Absolute accuracy is not state of the art. At matched input size, ViT-Adapter-L + M2F is still ahead (59.7 vs 58.3 PQ at $1280^2$); EoMT wins on the trade-off, not on the top number.
• Instance segmentation degrades most ($45.2$ vs $47.6$ AP at $640^2$), suggesting single-scale features hurt more for tight per-instance boundaries than for panoptic/semantic labelling.
• Mask annealing is a training-schedule hyperparameter (polynomial factor 0.9, staggered per block) with no principled derivation, and the failure mode when it is mishandled is severe ($27.4$ PQ when masking is dropped abruptly).
• The FPS advantages are measured on an H100 with FlashAttention-2 and torch.compile; part of the win is that plain ViTs are what current kernels are optimized for, so the margin is hardware- and toolchain-dependent (torch.compile had to be disabled for some baselines due to compilation errors, and full precision used for OneFormer with DiNAT-L).
• EoMT does not remove the mask module or the ${\mathbf{F}}_4$ upscaling, which remain the most compute-intensive non-ViT operations at inference.

See also

• SAM — the other “plain ViT plus a small head” segmentation design, with a heavyweight ViT encoder and a lightweight prompt-conditioned mask decoder
• SAM2 — extends that line to video
• DETR — origin of the learnable-query, set-prediction formulation that Mask Transformers and EoMT’s queries inherit
• Deformable DETR — supplies the multi-scale deformable attention used by ViT-Adapter and Mask2Former’s pixel decoder, the machinery EoMT deletes
• LW-DETR — the same “a plain ViT encoder is enough” argument, made for real-time detection
• Implementing Research Papers