Mask R-CNN

Mask R-CNN is a two-stage instance segmentation architecture.

• Paper: Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick (Facebook AI Research), Mask R-CNN, ICCV 2017, arXiv:1703.06870.
• Core idea: extends Faster R-CNN by adding a third branch, in parallel with the existing classification and box-regression branches, that predicts a binary segmentation mask for each Region of Interest (RoI).
• Headline result: state of the art on COCO instance segmentation, detection and keypoint estimation at once.
• Why it matters: its RoI-cropped mask head, refined through RoIAlign, is still the reference “two-stage” instance-segmentation design every later box-free method (YOLACT, SOLOv2) explicitly positions itself against.

Fig. 1 (paper): Faster R-CNN’s two stages (RPN → RoI feature extraction → classify/regress) are kept as-is; a small FCN mask branch runs in parallel with the existing class/box branch on the same per-RoI features.

Core idea: a mask branch in parallel, not in series

Mask R-CNN’s instance-first strategy is deliberately not novel: Faster R-CNN already has two outputs per candidate box — a class label and a box offset — from a Region Proposal Network (RPN) stage followed by RoI feature extraction, classification and box regression. Mask R-CNN adds a third branch, run in parallel with the class/box branch on the same per-RoI features, that outputs an object mask. This is a small fully convolutional network (FCN) applied to each RoI, predicting the mask in a pixel-to-pixel manner, rather than an fc-based approach as used by contemporaries (MNC, FCIS) that collapse the RoI into a vector before restoring spatial structure. The mask branch adds only a small computational overhead (~20% on typical models) since it reuses features Faster R-CNN already computes.

Two design decisions the paper identifies as critical to making this simple idea work well:

1. Decoupling mask and class prediction. Faster R-CNN’s classification branch already predicts the RoI’s class label; the mask branch predicts one binary mask per class ($K$ independent $m\times m$ masks, per-pixel sigmoid + binary loss) and simply uses the class branch’s own prediction to select which of the $K$ masks to output. This is unlike the common FCN convention of a per-pixel softmax over all classes at once, which forces per-pixel competition between classes. The paper shows this decoupling is a large factor in mask quality: switching to per-pixel multinomial/softmax masks (i.e., coupling class and mask prediction, as typical FCNs do) costs 5.5 mask AP. A class-agnostic mask head (single $m\times m$ mask regardless of class) is nearly as good as the class-specific version (29.7 vs. 30.3 AP on ResNet-50-C4), reinforcing that classification and segmentation are largely separable subtasks here.
2. RoIAlign, to fix a pixel-to-pixel misalignment problem RoIPool has that classification tolerates but mask prediction cannot.

RoIAlign

Fig. 3 (paper): the dashed grid is a feature map, the solid box an RoI with $2\times 2$ bins; dots are the 4 sampling points per bin, each computed by bilinear interpolation from the surrounding feature-map grid points.

RoIPool, the de facto operation Fast/Faster R-CNN use to extract a small fixed-size feature map (e.g. $7\times 7$) from each variable-sized RoI, quantizes twice: it rounds the floating-point RoI boundary to the feature map’s stride (computing $[x/16]$, a genuine integer rounding, for a stride-16 feature map), then rounds again when it subdivides the quantized RoI into bins. Both roundings shift every sampled feature by up to half a stride cell relative to the true RoI location. This has little effect on classification, which is robust to small translations, but a large effect on predicting pixel-accurate masks.

RoIAlign removes both quantization steps. The RoI is kept at floating-point precision throughout: it is divided into $k\times k$ bins without rounding, each bin is sampled at a fixed number of regularly-spaced points (4 in the paper’s implementation) whose value is computed by bilinear interpolation from the 4 nearest feature-map grid points, and the bin’s sampled points are aggregated (max or average — the paper finds average and max equivalent; it uses average). No aspect of the operation — the RoI’s own boundary, its subdivision into bins, or the sampling point locations — is ever quantized to the feature grid.

The gain is large and, per the ablations, entirely attributable to alignment rather than to bilinear sampling per se: RoIWarp (from the MNC paper), which adopts the same bilinear sampling idea as RoIAlign but still quantizes the RoI to the feature grid first, performs on par with plain RoIPool — confirming that quantization, not sampling method, was the actual defect. On a ResNet-50-C4 backbone (stride 16), RoIAlign improves mask AP by ~3 points over RoIPool, with most of the gain at high IoU (+5 AP${}_{75}$); on a deliberately larger-stride ResNet-50-C5 backbone (stride 32, where quantization error is worse), the gap widens to +7.3 mask AP and +10.5 AP${}_{75}$.

Head architecture

Fig. 4 (paper): both existing Faster R-CNN heads (ResNet-C4, FPN) are extended with a mask branch (bottom row of each panel) running in parallel with the pre-existing class/box branch (top row, grayed out as unchanged). All convs are $3\times 3$ except the final $1\times 1$ output conv; deconvs are $2\times 2$ stride 2; ReLU in all hidden layers.

For the ResNet-C4 backbone (RoI features from the 4th stage; classification uses ResNet’s 5th stage, res5, as its head), the mask branch takes the same $7\times 7\times 1024$ RoI features, but rather than sending them through the compute-intensive res5 (as the class/box branch does), it runs its own path: res5 → two $3\times 3$ convs → a $2\times 2$ stride-2 deconv (upsampling $7\times 7\to 14\times 14$) → a final $1\times 1$ conv down to $K$ channels, each a $14\times 14$ mask. For FPN, res5 is already folded into the backbone itself, so the head can be lighter: RoI features are $7\times 7\times 256$, and the mask branch is a stack of four $3\times 3$ convs, a $2\times 2$ stride-2 deconv to $14\times 14$, then more convs up to $28\times 28$ output resolution — twice the resolution of the C4 mask head, at lower compute, because FPN’s multi-scale RoI pooling already extracts more suitable features per RoI scale.

$m\times m$ is deliberately small ($14\times 14$ or $28\times 28$): the mask is later resized (bilinearly) to the RoI’s own size and binarized at a threshold of 0.5, so the mask head’s job is representing shape, not fine boundary detail, and a small fully convolutional output is sufficient and cheap because it never has to represent per-pixel detail beyond what the eventual upsampling recovers.

Hyperparameters

• Backbones evaluated: ResNet-50/101 and ResNeXt-50/101, both with a plain C4 head and with FPN.
• Loss: $L=L_{cls}+L_{box}+L_{mask}$, all three summed with equal weight, on each sampled RoI. $L_{mask}$ is defined only on positive RoIs and only on the ground-truth-class mask channel (other $K-1$ mask outputs do not contribute to the loss for that RoI).
• Training: image-centric, shorter side resized to 800px. Positive RoI threshold: IoU $\ge 0.5$ with a ground-truth box. 2 images/GPU, $N$ sampled RoIs per image with a 1:3 positive:negative ratio ($N=64$ for C4, $N=512$ for FPN). 8 GPUs (effective batch 16), 160k iterations, LR 0.02 decayed ×10 at 120k, weight decay 0.0001, momentum 0.9. ResNeXt: 1 image/GPU, same iteration count, LR 0.01.
• RPN: 5 scales, 3 aspect ratios; trained separately from Mask R-CNN by default (not sharing features) except where noted; shares the same backbone.
• RoIAlign: $k\times k$ bins per RoI (matching the head’s expected input resolution, e.g. $7\times 7$), 4 sampling points per bin, average pooling.
• Inference: 300 proposals (C4) or 1000 (FPN) from RPN → box branch → NMS → mask branch applied only to the top 100 scoring detections (not all proposals, unlike training) — this is faster and more accurate than running the mask branch on all proposals, because inference uses fewer, higher-quality RoIs than the densely-sampled positive/negative mix used in training.
• Runtime: ~200ms/frame on a GPU (ResNet-101-FPN, single-model); training takes 1–2 days on an 8-GPU machine.

Results

Instance segmentation mask AP on COCO test-dev (single-model):

Model	Backbone	AP	AP${}_{50}$	AP${}_{75}$	AP${}_S$	AP${}_M$	AP${}_L$
MNC (2015 winner)	ResNet-101-C4	24.6	44.3	24.8	4.7	25.9	43.6
FCIS+++ +OHEM (2016 winner)	ResNet-101-C5-dilated	33.6	54.5	—	—	—	—
Mask R-CNN	ResNet-101-C4	33.1	54.9	34.8	12.1	35.6	51.1
Mask R-CNN	ResNet-101-FPN	35.7	58.0	37.8	15.5	38.1	52.4
Mask R-CNN	ResNeXt-101-FPN	37.1	60.0	39.4	16.9	39.9	53.5

Mask R-CNN outperforms FCIS+++, the heavily-engineered 2016 challenge winner (multi-scale train/test, horizontal flip test, OHEM), without any of those additions. Backbone ablations (Table 2a) show the expected trend — deeper networks and FPN both help, ResNeXt improves further (30.3 → 32.7 → 34.3 → 36.7 AP going C4-R50 → C4-R101 → FPN-R101 → FPN-ResNeXt101) — with none of it dependent on architectural changes to the mask branch itself.

Mask R-CNN also wins on box AP using the exact same network (its box branch is an unmodified Faster R-CNN branch): the paper reports this as evidence that the multi-task training with the mask branch present acts as a beneficial regularizer/auxiliary signal for the shared backbone, not only as a segmentation add-on.

Limitations

• Speed. ~5 FPS on a GPU for the paper’s ResNet-101-FPN configuration — far from the real-time target the later box-free / one-stage lineage (YOLACT, SOLOv2) explicitly optimizes for. The mask branch itself is cheap; the cost is inherited from Faster R-CNN’s two-stage RPN → RoIAlign → per-RoI-head pipeline, which is inherently sequential (region proposals must exist before any per-instance computation can run).
• Mask resolution capped by RoI size. Because masks are predicted at a small, fixed resolution ($14\times 14$ or $28\times 28$) inside the RoI’s own coordinate frame and then upsampled, large-object mask boundaries are coarser than the object’s true resolution would allow — a limitation later addressed by SOLOv2’s unified high-resolution mask-feature representation and directly demonstrated in SOLOv2’s own qualitative comparison against Mask R-CNN’s $28\times 28$-capped masks.
• No shared computation across instances for mask prediction. Every RoI runs its own forward pass through the mask branch; a scene with many objects pays a roughly linear cost in the number of detections, unlike prototype-based (YOLACT) or dynamic-kernel (SOLOv2) heads whose per-image cost is closer to constant in the number of instances.

See also

• YOLACT — the real-time rebuttal: replaces RoI cropping and a per-instance FCN with a shared prototype dictionary and per-instance linear coefficients, explicitly to avoid Mask R-CNN’s sequential region-then-mask pipeline
• SOLOv2 — replaces RoI-relative mask prediction with dynamic per-instance convolution kernels applied densely, removing the bounding-box dependency this architecture still has
• Segmentation Head Comparison — this head placed side by side with YOLACT’s, SOLOv2’s, DETR’s and Mask2Former’s
• DETR — for a transformer-based detector, its panoptic segmentation extension plays a structurally similar role (a mask head bolted onto an existing detector’s per-object embeddings) but with an attention-map mechanism instead of RoIAlign
• DETR Family Comparison — Faster R-CNN’s transformer-era successors, for the box-detection side of this architecture