Mask2Former

Mask2Former (Masked-attention Mask Transformer) is a universal image segmentation architecture.

• Paper: Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov and Rohit Girdhar (Facebook AI Research / UIUC), Masked-attention Mask Transformer for Universal Image Segmentation, CVPR 2022, arXiv:2112.01527.
• Core idea: the first single architecture to outperform the best specialized architectures on all three standard segmentation tasks (semantic, instance and panoptic) on the same set of benchmarks, without changing the model, loss or training procedure between tasks; only the training data changes.
• Headline result: state of the art at the time with 57.8 PQ on COCO panoptic, 50.1 AP on COCO instance and 57.7 mIoU on ADE20K semantic segmentation.
• Efficiency: converges up to $6\times$ faster than its predecessor MaskFormer and cuts per-image training memory by $3\times$ (18 GB → 6 GB).

Fig. 1: universal architectures (Mask2Former, MaskFormer) vs. task-specialized SOTA (Max-DeepLab, Swin-HTC++, BEiT) across the three tasks. Mask2Former is the first universal model to lead on all three.

Motivation: mask classification as the universal formulation

Segmentation tasks differ only in what semantics define a segment — category membership for semantic segmentation, instance membership for instance segmentation, and both jointly for panoptic segmentation. Historically this led to disjoint architecture families: FCN-style per-pixel classification for semantic segmentation, and Mask R-CNN-style mask classification (predict a set of binary masks, each with a class label) for instance segmentation, each with its own specialized modules, losses, and hardware/training budgets — “duplicate research and (hardware) optimization effort.”

DETR showed that an end-to-end set-prediction objective built on learnable object queries and a Transformer decoder is general enough to reformulate any of these tasks as mask classification: predict $N$ binary masks plus $N$ class labels (one of which may be “no object”), regardless of whether the masks denote stuff regions, individual instances, or both. MaskFormer (the paper’s own direct predecessor) demonstrated this is competitive with specialized models on semantic and panoptic segmentation, but it lagged specialized instance segmentation architectures by more than 9 AP and required much longer training schedules (300 epochs, needing 32 GB just to fit a single image per GPU) than the specialized alternatives (e.g., Swin-HTC++ at 72 epochs). Mask2Former keeps MaskFormer’s meta-architecture — backbone → pixel decoder → Transformer decoder — but redesigns the Transformer decoder and the training recipe to close both gaps at once.

Architecture

Fig. 2: Mask2Former adopts MaskFormer’s meta-architecture (backbone, pixel decoder, Transformer decoder) but replaces standard cross-attention with masked attention, feeds one pixel-decoder resolution to each decoder layer round-robin, and reorders self-/cross-attention with learnable query features.

The meta-architecture has three components:

• Backbone — extracts low-resolution image features (ResNet or Swin-Transformer in the paper’s experiments).
• Pixel decoder — gradually upsamples the backbone’s low-resolution features into high-resolution per-pixel embeddings. Mask2Former is compatible with any pixel-decoder module (Table 4e ablates FPN, Semantic FPN, FaPN, BiFPN), but adopts a multi-scale deformable attention Transformer (MSDeformAttn) as its default: 6 MSDeformAttn layers applied to the backbone’s $1/8$, $1/16$ and $1/32$ feature maps, plus a simple upsampling layer with a lateral connection on the $1/32$ map to produce the final $1/4$-resolution per-pixel embedding used for mask prediction. MSDeformAttn was found to be the best-performing choice across all three tasks — different pixel decoders specialize (BiFPN for instance, FaPN for semantic), which the paper reads as evidence that Mask2Former, as a universal model, doubles as a good testbed for generalizable module design.
• Transformer decoder — operates on $N$ learnable object queries (100 by default), refining them across $L=3$ repeated 3-layer stages ($9$ layers total), and decoding a class label plus a binary mask from each final query. An auxiliary loss is attached to every intermediate decoder layer and to the learnable query features before they enter the decoder.

Its central proposal is a new decoder layer built around masked attention in place of the standard cross-attention MaskFormer/DETR use, plus an efficient multi-scale strategy to bring in high-resolution features without a large compute increase, and several free “optimization improvements” to the decoder layer ordering.

Masked attention

Standard cross-attention (with the residual path) at decoder layer $l$ computes

${\mathbf{X}}_l=\mathrm{softmax}({\mathbf{Q}}_l{\mathbf{K}}_l^{\mathsf{T}}){\mathbf{V}}_l+{\mathbf{X}}_{l-1}$

where ${\mathbf{X}}_l\in {\mathbb{R}}^{N\times C}$ are the $N$ $C$-dimensional query features at layer $l$, ${\mathbf{Q}}_l=f_Q({\mathbf{X}}_{l-1})$, and ${\mathbf{K}}_l,{\mathbf{V}}_l\in {\mathbb{R}}^{H_l{W}_l\times C}$ are the image features under linear transformations $f_K,f_V$ at spatial resolution $H_l\times W_l$. This attends every query over the entire feature map.

The paper’s diagnosis, following prior work on the slow convergence of DETR-style decoders, is that global context is exactly the problem: it takes many training epochs for cross-attention to learn to restrict itself to the relevant, localized object region. The hypothesis is that local features are sufficient to update query features, and that global context can instead be gathered through self-attention between queries. Masked attention implements this directly by constraining cross-attention to the foreground region of each query’s own predicted mask from the previous decoder layer:

${\mathbf{X}}_l=\mathrm{softmax}({\mathcal{M}}_{l-1}+{\mathbf{Q}}_l{\mathbf{K}}_l^{\mathsf{T}}){\mathbf{V}}_l+{\mathbf{X}}_{l-1}$

where the attention mask ${\mathcal{M}}_{l-1}$ at spatial location $(x,y)$ is

$${\mathcal{M}}_{l-1}(x,y)=\{\begin{array}{ll}0& \text{if }{\mathbf{M}}_{l-1}(x,y)=1\\ -\infty & \text{otherwise}\end{array}$$

${\mathbf{M}}_{l-1}\in \{0,1{\}}^{N\times H_l{W}_l}$ is the binarized mask prediction of the previous ($l-1$-th) decoder layer, thresholded at 0.5 and resized to the resolution of ${\mathbf{K}}_l$; ${\mathbf{M}}_0$ is the binary mask obtained from the initial query features ${\mathbf{X}}_0$ before the first decoder layer. Masked attention is a drop-in variant of cross-attention: it adds only the mask term ${\mathcal{M}}_{l-1}$ to the pre-softmax logits, at no extra computational cost, and Table 4a shows it is the single largest source of Mask2Former’s improvement over MaskFormer’s plain cross-attention across all three tasks — larger than the multi-scale high-resolution feature strategy. Table 4c also compares it against other cross-attention variants proposed concurrently for DETR (SMCA) and against K-Net’s mask-pooling cross-attention alternative; masked attention wins on all three tasks.

Multi-scale high-resolution features

High-resolution features improve segmentation quality, especially for small objects, but attending over a full high-resolution feature map at every decoder layer is expensive. Mask2Former instead builds a feature pyramid from the pixel decoder’s $1/32$, $1/16$ and $1/8$ resolution outputs (each augmented with a sinusoidal positional embedding and a learnable scale-level embedding) and feeds one resolution to one decoder layer at a time, in round-robin order from lowest to highest resolution, repeating the 3-layer pattern $L$ times for a total of $3L$ layers. This gives most of the accuracy gain from always using the highest-resolution map (naïve concatenation of all scales into every layer gives no further benefit — see Table 4d) while keeping FLOPs close to the single-scale baseline.

Optimization improvements

Three further changes to the standard Transformer decoder layer, each free of additional compute (Table 4b, ~1 PQ/AP/mIoU combined):

1. Switch the order of self- and cross-attention (i.e., masked attention before self-attention within a layer). The first self-attention layer operates on image-independent, zero-initialized query features with no signal from the image yet, so applying self-attention first is unlikely to enrich anything.
2. Make query features ${\mathbf{X}}_0$ learnable (previously zero-initialized in MaskFormer/DETR), supervised directly by the mask loss before being fed into the decoder to produce an initial mask prediction ${\mathbf{M}}_0$. These learnable queries turn out to behave like region-proposal-network proposals: before ever entering the decoder they already achieve a class-agnostic AR@100 of 50.3 on COCO val2017, rising to 56.8/57.4/57.7 after layers 3/6/9 of the decoder (Fig. 3), i.e. they carry usable localization signal from initialization.
3. Remove dropout entirely from the decoder — found to usually decrease performance rather than help.

Fig. 3: top — mask predictions of four selected learnable queries before the Transformer decoder ever sees them; bottom — class-agnostic AR@100 for the raw learnable queries vs. after decoder layers 3, 6 and 9.

Training-memory efficiency: point-based loss

MaskFormer’s main practical barrier was memory: computing the mask loss densely over the full-resolution mask required up to 18 GB per image, limiting training to a single image per 32 GB GPU. Motivated by PointRend and Implicit PointRend — which show a segmentation model can be trained with its mask loss calculated on $K$ randomly sampled points rather than the whole mask — Mask2Former samples points for both stages of training:

• Matching loss (building the bipartite-matching cost matrix): the same set of $K$ points is uniformly sampled for all predictions and all ground-truth masks.
• Final loss (between matched prediction/ground-truth pairs): different point sets are sampled per pair, using importance sampling (as in PointRend) to bias sampling toward uncertain regions.

$K=12,544$ points are used ($112\times 112$). This reduces per-image training memory by $3\times$, from 18 GB to 6 GB, without hurting accuracy — Table 5 shows point-sampled loss with point-sampled matching is not just cheaper but slightly better (47.3 vs. 45.9 mIoU on ADE20K) than mask-based matching, likely because uniform point sampling in the matching loss acts as a form of implicit regularization.

The mask loss itself combines binary cross-entropy and dice loss (rather than MaskFormer’s focal loss):

${\mathcal{L}}_{\text{mask}}={\lambda }_{\text{ce}}{\mathcal{L}}_{\text{ce}}+{\lambda }_{\text{dice}}{\mathcal{L}}_{\text{dice}},{\lambda }_{\text{ce}}={\lambda }_{\text{dice}}=5.0$

and the total per-query loss combines this with the classification loss, ${\mathcal{L}}_{\text{mask}}+{\lambda }_{\text{cls}}{\mathcal{L}}_{\text{cls}}$, with ${\lambda }_{\text{cls}}=2.0$ for queries matched to a ground-truth instance and $0.1$ for predictions matched to “no object.”

Hyperparameters

Transcribed from Sections 4.1–4.2.

Panoptic and instance segmentation (COCO)

Framework	Detectron2, following updated Mask R-CNN baseline settings
Optimizer	AdamW, step LR schedule
Initial learning rate	$0.0001$
Weight decay	$0.05$ (all backbones)
Backbone LR multiplier	$0.1$
LR decay	$\times 0.1$ at 0.9 and 0.95 fractions of total training steps
Epochs (default)	50
Batch size	16
Augmentation	large-scale jittering (LSJ), scale sampled from $[0.1,2.0]$, fixed crop to $1024\times 1024$
Inference (standard Mask R-CNN setting)	resize shorter side to 800, longer side up to 1333
Transformer decoder	$L=3$ (9 layers total), 100 queries by default
Pixel decoder	6 MSDeformAttn layers on $1/8$, $1/16$, $1/32$ feature maps; upsampling layer w/ lateral connection for $1/4$ output
Mask loss	${\lambda }_{\text{ce}}=5.0$, ${\lambda }_{\text{dice}}=5.0$ (BCE + dice, not focal loss)
Classification loss weight	${\lambda }_{\text{cls}}=2.0$ (matched), $0.1$ (“no object”)
Point sampling	$K=12,544$ points ($112\times 112$); uniform for matching loss, importance sampling for final loss

Semantic segmentation (ADE20K)

Same as above except: (1) the $0.1\times$ backbone LR multiplier is applied to both CNN and Transformer backbones (not just CNN as in MaskFormer); (2) both ResNet and Swin backbones use an initial learning rate of $0.0001$ and weight decay $0.05$, instead of MaskFormer’s differing learning rates per backbone type.

Results

Panoptic segmentation, COCO val2017 (Table 1). With a ResNet-50 backbone and 50 epochs, Mask2Former reaches 51.9 PQ, versus MaskFormer’s 46.5 PQ (300 epochs) — a 5.4 PQ gain while converging 6$\times$ faster. With Swin-L (200 epochs, 100 queries doubled to account for scale), Mask2Former sets 57.8 PQ, ahead of MaskFormer’s 52.7 PQ and concurrent K-Net’s 54.6 PQ. It also reports higher ${\text{AP}}_{\text{pan}}^{\text{Th}}$ (AP evaluated on “thing” categories using instance-segmentation annotations) and ${\text{mIoU}}_{\text{pan}}$ (semantic mIoU derived from the same panoptic-only-trained model) than DETR and MaskFormer — evidence the model transfers to other tasks even when trained on panoptic annotations alone.

Instance segmentation, COCO val2017 (Table 2). With ResNet-50, Mask2Former reaches 43.7 AP against a strong Mask R-CNN + LSJ baseline’s 42.5 AP, using $8\times$ fewer training iterations (50 vs. 400 epochs). With Swin-L, it reaches 50.1 AP, competitive with the specialized Swin-HTC++ (50.2 AP) while improving Boundary AP by 2.1 points, reflecting the benefit of high-resolution mask prediction. Performance on small objects (AP${}^S$) still trails specialized architectures — the paper’s own noted headroom.

Semantic segmentation, ADE20K val (Table 3). Mask2Former consistently outperforms MaskFormer across backbones (e.g. R50: 47.2 vs. 44.5 mIoU single-scale). With Swin-L-FaPN as pixel decoder, it sets 57.7 mIoU, ahead of the specialized BEiT-L (57.0 mIoU).

Generalization (Cityscapes, Table 6; also Mapillary Vistas, ADE20K in the appendix). Mask2Former (Swin-L) reaches 66.6 PQ, 43.6 AP${}_{\text{pan}}^{\text{Th}}$, 82.9 mIoU${}_{\text{pan}}$ on Cityscapes val, competitive with or ahead of specialized models like Panoptic-DeepLab and SegFormer.

Ablations

All on ResNet-50, testing all three tasks together (Table 4):

• Masked attention vs. high-resolution features (4a): removing masked attention costs $-5.9$ AP / $-4.8$ PQ / $-1.7$ mIoU; removing the efficient multi-scale strategy costs $-2.2$ AP / $-1.7$ PQ / $-1.1$ mIoU. Masked attention is the larger contributor on all three tasks.
• Optimization improvements (4b): removing learnable query features, cross-attention-first ordering, and dropout individually cost small amounts (up to $-1.4$ AP combined), stacking to $-1.4$ AP / $-1.1$ PQ / $-0.9$ mIoU when all three are removed together — free accuracy at no extra compute.
• Masked attention vs. alternatives (4c): beats plain cross-attention, SMCA, and K-Net’s mask-pooling cross-attention on all three tasks.
• Feature resolution (4d): single highest-resolution scale ($1/8$) matches the efficient multi-scale strategy’s accuracy at much higher FLOPs (239G → 226G at equal accuracy); naïve concatenation of all scales into every layer gives no further gain over round-robin.
• Pixel decoder choice (4e): MSDeformAttn is consistently the best performer across all three tasks, ahead of FPN, Semantic FPN, FaPN and BiFPN — though the latter two are respectively strongest for their specific task (instance, semantic), underscoring that no single pixel decoder generalizes as well as MSDeformAttn.
• Point loss vs. mask loss (Table 5): switching both matching and final loss to sampled points cuts training memory $3\times$ (18 GB → 6 GB per image on COCO) while improving accuracy over mask-based loss (47.3 vs. 45.9 mIoU on ADE20K).

Limitations

The paper is explicit about three:

1. Task-specific training is still required. Table 7 shows a single Mask2Former trained only on panoptic annotations performs slightly worse than the same architecture trained specifically for instance or semantic segmentation with matching annotations, across all three tested datasets. The universal architecture does not yet give a universal model — training data must still match the target task.
2. Small objects remain difficult. Despite the multi-scale strategy, Mask2Former still lags specialized architectures on AP${}^S$ (Table 2) and on high-resolution/small-object mIoU (Table 3). The authors attribute this to insufficient utilization of the feature pyramid and to losses not specifically designed for small objects, both left as future work — along with dilated backbones as used in DETR.
3. No single model across datasets. The paper’s stated future goal is a model trainable once for multiple tasks and multiple datasets simultaneously; Mask2Former as presented must still be retrained per dataset/task combination.

See also

• DETR — originates the learnable-query, set-prediction, Hungarian-matching formulation that MaskFormer and Mask2Former’s Transformer decoder descend from.
• Your ViT is Secretly an Image Segmentation Model (EoMT) — argues that, given a sufficiently large and sufficiently pre-trained plain ViT, Mask2Former’s entire pixel decoder plus Transformer decoder (the two components this article covers) is largely redundant: EoMT strips both out and instead concatenates queries directly into ViT blocks, reproducing masked attention’s convergence benefit via mask annealing during training and discarding it at inference.
• RefT — builds its few-shot instance segmentation baseline directly on a Mask2Former meta-architecture (same pixel decoder, same Transformer decoder with masked attention), adding a dual support-branch reference mechanism on top; RefT’s own ablations confirm masked attention and the shared MSDeformAttn pixel decoder as inherited strengths.
• SAM, SAM2, SAM3 — promptable segmentation foundation models built on a different (prompt-conditioned, not closed-vocabulary mask-classification) paradigm, but converging on the same “queries cross-attend into image features to produce masks” mechanism at the decoder level.
• Segmentation Head Comparison — this mask-classification head placed side by side with Mask R-CNN’s, YOLACT’s, SOLOv2’s and DETR’s earlier panoptic head
• Mask R-CNN, YOLACT, SOLOv2 — the RoI-based, prototype-based and dynamic-kernel instance segmentation heads Mask2Former’s single architecture matches or beats across all three segmentation tasks at once
• Implementing Research Papers