A similar model: YOLOv8

Segmentation Head Comparison

A side-by-side comparison of five ways to turn a detector’s per-object representation into a pixel mask: Mask R-CNN’s RoI-cropped FCN head, YOLACT’s prototype + coefficient decomposition, SOLOv2’s dynamic per-location convolution, DETR’s attention-map panoptic head, and Mask2Former’s mask-classification Transformer decoder. All five answer the same question — given that the model has decided “there is an object here,” how does it produce that object’s mask? — with structurally different answers, at different points in the trade-off between per-instance compute, mask resolution, and dependence on a bounding box.

The mechanism, side by side

Every head below takes some per-object representation (an RoI, a coefficient vector, a dynamic kernel, an attention query) and some shared spatial feature map, and combines them into a mask. The diagrams are original for this wiki (not reproductions of paper figures) redrawn in one visual style so the five mechanisms can be compared directly; see each model’s own article for the paper’s own figures and the full derivation.

Mask R-CNN — crop, then predict

RoIAlign crops backbone features into the RoI’s own coordinate frame at a small, fixed resolution ($7\times 7$), and a small FCN (a few convs + one deconv) predicts a per-class mask from that crop — one full forward pass through the mask head per detected instance. The mask never sees anything outside its own (possibly imprecise) box.

YOLACT — prototypes, weighted per instance

A single shared dictionary of $k$ full-image prototype masks is produced once per image; each detected instance gets only a $k$-length coefficient vector, and its mask is the linear combination $\sigma (P{C}^{\mathsf{T}})$ — one matrix multiply for all instances at once, then a crop to the predicted box. See YOLACT for the full mechanism and why the crop is not what actually localizes instances.

SOLOv2 — a kernel per grid cell, convolved densely

Like YOLACT, feature production and per-instance combination are decoupled — but here the per-instance side is a dynamic convolution kernel (predicted by a kernel branch, conditioned on which $S\times S$ grid cell an object’s center falls in) rather than a coefficient vector, convolved with a shared, unified, image-wide mask feature map. No bounding box is predicted or needed anywhere in the pipeline; SOLOv2’s box is a byproduct computed from the mask, not the other way around.

DETR panoptic head — attention maps upsampled by an FPN

Each of DETR’s $N$ decoder object queries already carries a learned embedding; a bespoke multi-head attention module reuses it to attend over the encoder’s memory, producing $M$ low-resolution attention heatmaps per object, which an FPN-style CNN (with lateral connections to intermediate backbone features, structurally similar to Mask R-CNN’s own FPN) upsamples into a full mask logit map. Unlike every other head here, this one is layered on top of an existing box-prediction requirement — DETR’s Hungarian matching needs box distances to exist before the mask head can even be trained.

Mask2Former — dot product with a per-pixel embedding

A pixel decoder produces one per-pixel embedding map for the whole image; a Transformer decoder produces, per query, a class label and a mask embedding vector. A query’s mask is simply the dot product of its mask embedding against every pixel’s embedding, thresholded — no crop, no RoI, no prototype dictionary, no convolution kernel, and (unlike DETR panoptic) no box prediction anywhere in the model. See Mask2Former for the masked-attention mechanism that makes this converge fast enough to be practical.

Taxonomy

	Per-instance representation	Shared spatial component	Combined by	Needs a box?
Mask R-CNN	cropped RoI features	— (cropped per-instance, not shared)	small FCN, run per RoI	Yes — RoI is the box
YOLACT	$k$-length coefficient vector	$k$ prototype masks (protonet)	linear combination + sigmoid	Only to crop after assembly
SOLOv2	dynamic conv kernel ($S\times S$ grid)	unified mask feature map	convolution	No — box is a byproduct
DETR (panoptic)	decoder query embedding	encoder memory + backbone features	multi-head attention + FPN-style CNN	Yes — trained jointly with box head
Mask2Former	mask embedding vector	per-pixel embedding map	dot product	No

Two independent axes explain most of the design space: is there a fixed spatial crop involved (Mask R-CNN and, weakly, YOLACT’s post-hoc crop) or does the head operate over the full image at once (SOLOv2, Mask2Former, and DETR’s encoder-attention step), and does a bounding box have to exist for the mask to exist (Mask R-CNN, DETR panoptic) or is the mask the primary representation with the box, if any, derived from it (YOLACT’s crop box already existed independently; SOLOv2 and Mask2Former need no box at all).

Why the AP numbers below don’t compare cleanly

As with DETR Family Comparison, put these five numbers next to each other with real caution:

1. Different tasks. Mask R-CNN, YOLACT and SOLOv2’s headline numbers are COCO instance segmentation AP. DETR’s headline mask-head number is COCO panoptic PQ (things + stuff together) — its things-only mask AP is reported mainly to show the box→mask deficit, not as a competitive instance-segmentation claim. Mask2Former’s headline is also panoptic PQ, but it additionally reports competitive instance AP on the same weights, unlike DETR.
2. Different backbones and schedules. ResNet-50 vs. ResNet-101 vs. ResNeXt vs. Swin-based backbones are mixed below; deeper/transformer backbones are worth several AP on their own, independent of the mask head.
3. Speed numbers use different GPUs. YOLACT reports Titan Xp; SOLOv2 and Mask R-CNN (as re-measured by SOLOv2) use a V100; Mask2Former reports neither FPS nor a comparable real-time claim, since it was not designed as a real-time architecture. Treat the FPS column as a rough ordering only, not a precise ratio.
4. DETR panoptic PQ${}^{th}$ is not the same metric as instance-segmentation mask AP. A PQ number already folds in the model’s own detection recall/precision at its own confidence threshold; it is not reported on the same scale as AP and cannot be read as a row in the same ranking.

COCO numbers, with the above caveats

Instance segmentation, test-dev, single model, no test-time augmentation, closest published backbone to ResNet-50/101-FPN for each:

Head	Model	Backbone	Mask AP	FPS (GPU)	Needs NMS?
RoI-cropped FCN	Mask R-CNN	ResNet-101-FPN	35.7	~5 (unspecified GPU)	Yes (box NMS)
Prototype + coefficient	YOLACT-550	ResNet-101-FPN	29.8	33.5 (Titan Xp)	Yes (Fast NMS)
Dynamic kernel + feature map	SOLOv2	ResNet-50-FPN	38.8	18 (V100)	Yes (Matrix NMS)
Dynamic kernel + feature map (real-time)	SOLOv2 (speed-priority)	ResNet-50-FPN	34.0	46.5 (V100)	Yes (Matrix NMS)
Mask-classification Transformer	Mask2Former	ResNet-50	43.7	9.7 (V100)	No (queries are already disjoint by construction, no box NMS step)

Panoptic segmentation, COCO val2017, ResNet-101-class backbones (Mask2Former also scales much further on a Swin-L backbone — 57.8 PQ — included below only to show the ceiling, not for a same-backbone comparison):

Head	Model	PQ	PQ${}^{th}$	PQ${}^{st}$
Attention map + FPN-style CNN	DETR-R101	45.1	50.5	37.0
Dynamic kernel + feature map	SOLOv2 (ResNet-50-FPN)	42.1	49.6	30.7
Mask-classification Transformer	Mask2Former-R101	52.6	58.5	43.7
Mask-classification Transformer (ceiling)	Mask2Former-Swin-L	57.8	64.2	48.1

Only SOLOv2 and Mask2Former report a number in both tables on comparable-ish terms (SOLOv2: 38.8 instance AP and 42.1 PQ on the same ResNet-50-FPN backbone; Mask2Former: 43.7 instance AP and 51.9 PQ, both on ResNet-50), which is itself informative — instance and panoptic performance do not rank models the same way, and a head’s PQ number tells you little about its instance AP without checking both.

NMS dependence

All five heads except Mask2Former still rely on some form of non-maximum suppression to remove duplicate detections of the same object:

• Mask R-CNN: ordinary box NMS on the detection branch, before the mask branch is even run (masks are only computed for the top 100 surviving boxes).
• YOLACT: introduces Fast NMS, a fully parallel relaxation of standard NMS, motivated purely by the fact that sequential NMS becomes a real bottleneck once the rest of the pipeline is fast enough to be real-time.
• SOLOv2: introduces Matrix NMS, independently derived from a Soft-NMS-style score-decay view rather than YOLACT’s relaxation, and empirically both faster and 0.4 AP better than Fast NMS.
• DETR / Mask2Former: neither needs NMS in the classic sense — DETR’s bipartite matching loss already trains the model not to produce duplicate box predictions for one object, and Mask2Former’s set of queries is trained the same way. Mask2Former’s final panoptic merge is a pixel-wise argmax over queries (as DETR’s panoptic head also uses), not an IoU-threshold suppression step — a structurally different way of resolving overlaps that has no threshold to tune.

Which one to use

• Best mask quality at a given compute budget, task-agnostic — Mask2Former: same architecture, weights, and loss across semantic/instance/panoptic tasks, state of the art on all three at publication, at the cost of not targeting real-time inference at all.
• Real-time instance segmentation on a GPU — SOLOv2’s speed-priority variant if you can retrain from scratch (its dynamic-kernel design has no box dependency and the best mask-detail/speed trade-off of the real-time options); YOLACT if you specifically want the prototype representation’s compressibility and interpretability (Figure 5 of the YOLACT paper), or want the head design YOLOv8’s -seg checkpoints already ship.
• Maximum boundary fidelity on large objects — SOLOv2, by a wide margin on AP${}_L$, because its unified mask-feature representation is never resolution-capped the way Mask R-CNN’s RoI-relative $28\times 28$ output is.
• You already have a Faster R-CNN-family two-stage detector and want to add masks with minimal architecture change — Mask R-CNN: the mask branch is genuinely a drop-in addition, at the cost of being the slowest option here and capping mask resolution to whatever the RoI head uses.
• You already have a DETR-family detector and want panoptic output — the DETR panoptic head is the smallest addition (freeze the existing box model, train only the mask head for 25 epochs), though Mask2Former is the better target if you are willing to change the decoder itself.

See also

• Mask R-CNN, YOLACT, SOLOv2, DETR, Mask2Former — the five architectures compared here
• DETR Family Comparison — the same kind of side-by-side comparison, for detection rather than segmentation heads, including the same caveats about cross-paper numbers
• Few-Shot Segmentation Comparison — a similar comparison one level up the task hierarchy: what changes when segmentation has to generalize to unseen classes from a handful of examples
• YOLOv8 — ships YOLACT’s prototype head essentially unchanged as its -seg variant
• Your ViT is Secretly an Image Segmentation Model — argues a large enough plain ViT needs none of these dedicated head designs at all