DETR

DETR (DEtection TRansformer)

DETR is an object detection model that reformulates detection as a direct set prediction problem.

• Paper: Carion, Massa, Synnaeve, Usunier, Kirillov, Zagoruyko (Facebook AI Research), End-to-End Object Detection with Transformers, ECCV 2020, arXiv:2005.12872.
• Core idea: removes the hand-designed components — anchor boxes, region proposals, non-maximum suppression (NMS) — that earlier detectors (Faster R-CNN, YOLO, RetinaNet) relied on.

Core idea

Traditional detectors predict a large number of candidate boxes and then use surrogate tasks (anchor assignment, NMS) to reduce them to a final set. DETR instead:

1. Predicts a fixed-size set of N detections in parallel (e.g., N = 100), independent of image content.
2. Trains with a loss that performs bipartite matching between predictions and ground-truth objects, so each prediction is responsible for at most one object (or “no object”).

This makes the pipeline end-to-end differentiable with no post-processing step required at inference beyond a simple confidence threshold.

Architecture

DETR combines a CNN backbone with a standard Transformer encoder-decoder:

• CNN backbone (e.g., ResNet-50) extracts a lower-resolution feature map from the input image.
• Flatten + positional encoding: the feature map is flattened into a sequence of tokens and combined with fixed sinusoidal positional encodings (since the transformer itself is permutation-invariant).
• Transformer encoder: standard multi-head self-attention layers process the flattened image features, letting each location attend to all others — useful for reasoning about relationships between distant objects.
• Transformer decoder: takes N learned object queries (fixed, learned embeddings, one per “slot”) and attends to the encoder output via cross-attention, transforming each query into an output embedding. Unlike autoregressive transformers, all N queries are decoded in parallel.
• Prediction heads: each decoder output embedding is passed through a shared feed-forward network (FFN) to predict a class label (including a special “no object” / ∅ class) and a bounding box (center x, y, width, height, normalized).

Bipartite matching loss

Because the N predictions are unordered, DETR needs a way to assign each ground-truth object to exactly one prediction before computing a loss. It finds the optimal bipartite matching σ between predictions and ground truth (padded with ∅ to size N) that minimizes a total matching cost, solved efficiently with the Hungarian algorithm.

The matching cost per pair combines:

• Class prediction cost (negative predicted probability of the correct class)
• Bounding box cost, using a mix of L1 loss and generalized IoU (GIoU) loss (L1 alone is scale-dependent and penalizes large boxes disproportionately)

Once matched, the training loss is the sum of:

• Negative log-likelihood for class prediction (with down-weighted loss for the ∅ class, since most predictions are unmatched)
• L1 + GIoU loss on box coordinates, only for matched pairs

This is the key trick that lets DETR train without NMS: since each ground-truth object is matched to a unique prediction, the model is never rewarded for producing duplicate boxes for the same object.

Auxiliary losses

Since predictions are also produced at intermediate decoder layers, DETR applies the same matching + loss at every decoder layer (auxiliary losses), which speeds up convergence and improves accuracy.

Panoptic segmentation head

Fig. 8 (paper): each of the $N$ decoder box embeddings attends (via $M$-head attention) over the encoder’s output memory, producing $M$ small-resolution attention heatmaps per object; an FPN-style CNN upsamples and concatenates these with intermediate ResNet feature maps (Res2–Res5) to produce one mask logit map per object at stride 4, and a final pixel-wise argmax across objects merges them into one panoptic segmentation map.

Similarly to how Faster R-CNN extends to Mask R-CNN, DETR extends naturally to panoptic segmentation (Kirillov et al., 2019 — jointly segmenting countable “things” and uncountable “stuff” like sky or grass) by adding a mask head on top of the existing decoder outputs, treating both kinds of classes in one unified formulation. DETR is first trained to predict boxes around both stuff and things categories on COCO’s panoptic annotations (53 stuff + 80 thing categories) using the ordinary box recipe — bipartite matching needs box distances to be computable, so box prediction remains a requirement even though boxes are discarded for stuff classes at the end.

Mechanism (from the official implementation, models/segmentation.py, since the architectural specifics beyond Fig. 8 are only in the paper’s supplement):

1. MHAttentionMap — a bespoke multi-head attention module that, unlike ordinary cross-attention, returns only the attention weights (no value-weighted sum). For each of the $N$ final decoder embeddings (queries) as $Q$ and the encoder’s output memory as $K$, it computes per-head attention maps via $\mathrm{softmax}\left(\frac{Q{K}^{\mathsf{T}}}{\sqrt{d/n_{heads}}}\right)$, masked at padded image positions, giving an $N\times M\times H/32\times W/32$ tensor — $M$ low-resolution attention heatmaps per object.
2. MaskHeadSmallConv — an FPN-style upsampling CNN. It concatenates each object’s $M$ attention maps with the projected encoder features, then progressively upsamples ($\times 2$ bilinear + $3\times 3$ conv + GroupNorm + ReLU, four times) while at each stage fusing in a lateral connection from an intermediate ResNet feature map (Res3, Res2 — analogous to Mask R-CNN/FPN’s lateral connections), finishing at a single-channel mask logit per object at stride 4 ($N\times H/4\times W/4$).
3. Each mask is supervised independently with DICE/F-1 loss and Focal loss (no competition between objects during training, exactly as Mask R-CNN’s per-class sigmoid masks avoid competition between classes).
4. At inference, the final panoptic map is produced by a pixel-wise argmax across all predicted object masks — every pixel is assigned to whichever object’s mask logit is highest there, guaranteeing non-overlapping final segments with no separate heuristic needed to reconcile conflicting masks (unlike e.g. Panoptic FPN, which needs one).

Training regime: the mask head can be trained jointly with the box predictor or in two steps (train DETR for boxes only, freeze all weights, then train only the mask head for 25 epochs); the paper reports similar results either way and uses the two-step variant since it has a shorter total wall-clock training time.

On COCO val, DETR-R101 reaches 45.1 PQ (50.5 PQ${}^{th}$ on things, 37.0 PQ${}^{st}$ on stuff), just ahead of the paper’s own strong PanopticFPN++ baseline (44.1 PQ; 51.0 PQ${}^{th}$, 33.6 PQ${}^{st}$) trained with the same data augmentation for fair comparison. The gap is concentrated entirely on stuff classes (37.0 vs. 33.6), which the paper attributes to the encoder’s global self-attention being particularly suited to reasoning about large, spatially diffuse stuff regions; on things classes DETR is merely competitive, marginally behind PanopticFPN++ (50.5 vs. 51.0 PQ${}^{th}$) despite a deficit of up to 8 points on the things-only mask-AP metric computed before the pixel-wise-argmax post-processing (33.0 vs. 39.7 AP for the R101 pair).

Results and characteristics

• On COCO, DETR achieves accuracy comparable to a well-tuned Faster R-CNN baseline, with a simpler pipeline.
• Strengths: much better performance on large objects (global self-attention helps reason about the whole image); simpler architecture with no hand-crafted anchors or NMS; the encoder attention maps are often interpretable, separating individual object instances.
• Weaknesses: notably worse performance on small objects; very slow to train — the original model required 500 epochs to converge, largely because the bipartite matching assignment is unstable early in training; quadratic complexity of self-attention over image feature maps is expensive for high-resolution inputs.

Hyperparameters

As reported in the original paper for the baseline COCO models:

Architecture

• Backbone: ResNet-50 or ResNet-101 (ImageNet-pretrained), dilated-C5 variant (DC5) also explored for higher-resolution features
• Transformer: 6 encoder layers, 6 decoder layers
• Hidden dimension (d_model): 256
• Attention heads: 8
• Feed-forward network dimension: 2048
• Dropout: 0.1 (applied in transformer layers)
• Number of object queries (N): 100

Optimization

• Optimizer: AdamW
• Learning rate (transformer): 1e-4
• Learning rate (backbone): 1e-5
• Weight decay: 1e-4
• Batch size: 64 images (across 16 GPUs, 4 images/GPU)
• Training length: 300 epochs (500 epochs in the original/slower schedule), with learning rate dropped by 10× after 200 epochs (or 400 for the 500-epoch schedule)
• Gradient clipping: max norm 0.1

Loss weights

• Classification loss weight: 1
• L1 bounding box loss weight (λ_L1): 5
• GIoU loss weight (λ_giou): 2
• No-object (∅) class weight: 0.1 (down-weighted to counteract class imbalance, since most of the N slots are unmatched)

Data augmentation

• Scale augmentation: resize so the shortest side is between 480 and 800 pixels, longest side ≤ 1333
• Random crop augmentation
• Random horizontal flip

Influence

DETR’s slow convergence and weak small-object performance motivated a wave of follow-up work, most notably Deformable DETR, which replaces full self/cross-attention with deformable (sparse, learned-offset) attention over a small set of key sampling points, drastically speeding up training and improving small-object detection. Later variants (DINO, DAB-DETR, DN-DETR, Conditional DETR, etc.) further refined query design and training stability. DETR is generally credited with establishing the transformer-based, NMS-free “set prediction” paradigm as a viable alternative to the anchor-based detection pipeline that had dominated the field since Faster R-CNN.

See also

• Deformable DETR — fixes DETR’s slow convergence and weak small-object performance with sparse multi-scale deformable attention
• Conditional DETR — fixes the slow convergence by reformulating the decoder cross-attention query, keeping dense attention
• Mask2Former — the later, unified mask-classification architecture that generalizes this panoptic head’s set-prediction idea to semantic, instance and panoptic segmentation alike, without needing a separate box-prediction requirement
• Mask R-CNN — the analogous “bolt a mask branch onto an existing detector” extension for Faster R-CNN, using RoIAlign instead of an attention-map + FPN-style mask head
• Segmentation Head Comparison — this panoptic head placed side by side with Mask R-CNN’s, YOLACT’s, SOLOv2’s and Mask2Former’s mask heads
• All Articles