Deformable DETR

Deformable DETR is an end-to-end object detector that replaces dense Transformer attention in DETR with deformable attention.

• Paper: Zhu, Su, Lu, Li, Wang and Dai (SenseTime Research / USTC / CUHK), Deformable DETR: Deformable Transformers for End-to-End Object Detection, ICLR 2021, arXiv:2010.04159.
• Kept from DETR: the set-prediction formulation — Hungarian matching, no NMS, no anchors.
• Core idea: replaces the dense Transformer attention over feature-map pixels with a (multi-scale) deformable attention module that attends to only a small fixed set of learned sampling points around a reference point.
• Headline result: removes both of DETR’s practical defects at once: it converges in 50 epochs instead of 500 ($10\times$ fewer) and it can consume high-resolution multi-scale feature maps, lifting small-object AP substantially.

Motivation: why DETR is slow and weak on small objects

Both problems trace back to the same source — Transformer attention treating every feature-map pixel as a key element.

• Slow convergence. With standard initialization the attention logits are near-uniform, so $A_{mqk}\approx 1/N_k$ at the start of training. In the image domain $N_k=HW$ is huge, so the gradients w.r.t. individual keys are ambiguous and it takes a very long schedule for the attention maps to sharpen onto the few meaningful locations they end up using. DETR needs 500 epochs on COCO, roughly 10–20× slower than Faster R-CNN.
• Limited spatial resolution. Encoder self-attention has complexity $O(N_q{C}^2+N_k{C}^2+N_q{N}_{k}C)$; with $N_q=N_k=HW\gg C$ the third term dominates, i.e. $O(H^2{W}^{2}C)$ — quadratic in the spatial size. High-resolution or multi-scale feature maps are therefore unaffordable, and small objects suffer (DETR: ${\text{AP}}_S=20.5$ vs Faster R-CNN’s $26.6$).

Deformable convolution (Dai et al., 2017) already attends to sparse spatial locations efficiently, but lacks the element-relation modelling that makes Transformers work. Deformable attention is the combination of the two: sparse, data-dependent sampling as a pre-filter for prominent key elements, with attention weights over the sampled points.

Deformable attention module

Given an input feature map $\mathbf{x}\in {\mathbb{R}}^{C\times H\times W}$, a query element $q$ with content feature ${\mathbf{z}}_q$ and a 2-d reference point ${\mathbf{p}}_q$:

$\text{DeformAttn}({\mathbf{z}}_q,{\mathbf{p}}_q,\mathbf{x})={\sum }_{m=1}^M{\mathbf{W}}_m[{\sum }_{k=1}^K{A}_{mqk}\cdot {\mathbf{W}}_m^{\prime }\mathbf{x}({\mathbf{p}}_q+\Delta {\mathbf{p}}_{mqk})]$

where:

• $m\in \{1,\dots ,M\}$ — index of the attention head; each head samples its own points and has its own weights.
• $k\in \{1,\dots ,K\}$ — index of the sampled point within a head. $K\ll HW$ (in practice $K=4$), so a query looks at a handful of locations instead of all $HW$ pixels.
• ${\mathbf{p}}_q$ — the query’s reference point, a 2-d location in the feature map: where this query starts looking.
• $\Delta {\mathbf{p}}_{mqk}\in {\mathbb{R}}^2$ — the sampling offset, an unconstrained real-valued 2-d shift predicted for point $k$ of head $m$. It says “look this far from the reference point”.
• $\mathbf{x}({\mathbf{p}}_q+\Delta {\mathbf{p}}_{mqk})$ — the feature read out at that shifted location. The location is fractional, so this is a bilinear interpolation of the four surrounding pixels (which also makes the offsets differentiable).
• $A_{mqk}\in [0,1]$ — the attention weight for that sampled point, with ${\sum }_{k=1}^K{A}_{mqk}=1$ per head: how much this particular sample contributes.
• ${\mathbf{W}}_m^{\prime }\in {\mathbb{R}}^{C_v\times C}$ — value projection of head $m$, mapping the sampled feature into the head’s $C_v=C/M$-dimensional subspace.
• ${\mathbf{W}}_m\in {\mathbb{R}}^{C\times C_v}$ — output projection of head $m$, mapping the head’s result back to $C$ channels before the heads are summed.

Read from the inside out, the formula is therefore: for each head, sample $K$ points at learned offsets around the reference point, take a weighted average of their (projected) features, project the result back, and sum over heads. It has exactly the shape of ordinary multi-head attention — the only change is that the inner sum runs over $K$ chosen locations rather than over all $N_k$ key elements.

The crucial structural difference from standard attention: both the offsets and the attention weights are predicted directly from the query feature ${\mathbf{z}}_q$, not from query–key dot products. A single linear projection maps ${\mathbf{z}}_q$ to $3MK$ channels — the first $2MK$ are the offsets $\Delta {\mathbf{p}}_{mqk}$, the remaining $MK$ go through a softmax to give $A_{mqk}$. No key is ever touched to decide whether to attend to it, which is what removes the $N_q{N}_k$ term.

Multi-scale deformable attention

The module extends to $L$ feature levels $\{{\mathbf{x}}^l{\}}_{l=1}^L$ with essentially no extra machinery. With normalized reference-point coordinates ${\hat{\mathbf{p}}}_q\in [0,1{]}^2$ (where $(0,0)$ and $(1,1)$ are the top-left and bottom-right image corners):

$\text{MSDeformAttn}({\mathbf{z}}_q,{\hat{\mathbf{p}}}_q,\{{\mathbf{x}}^l{\}}_{l=1}^L)={\sum }_{m=1}^M{\mathbf{W}}_m[{\sum }_{l=1}^L{\sum }_{k=1}^K{A}_{mlqk}\cdot {\mathbf{W}}_m^{\prime }{\mathbf{x}}^l({\varphi }_l({\hat{\mathbf{p}}}_q)+\Delta {\mathbf{p}}_{mlqk})]$

where the new pieces relative to the single-scale version are:

• $l\in \{1,\dots ,L\}$ — index of the feature level; the inner sum now runs over levels and sampling points.
• ${\hat{\mathbf{p}}}_q\in [0,1{]}^2$ — the reference point in image-normalized coordinates, so one point addresses all levels regardless of their resolutions.
• ${\varphi }_l({\hat{\mathbf{p}}}_q)$ — rescales that normalized coordinate to the pixel grid of level $l$ (multiply by $H^l,W^l$).
• $\Delta {\mathbf{p}}_{mlqk}$, $A_{mlqk}$ — offset and weight for point $k$ at level $l$ in head $m$; the weights are now normalized jointly across levels, ${\sum }_l{\sum }_k{A}_{mlqk}=1$, so a head can shift its budget toward whichever scale is informative.

Each query now samples $LK$ points spread across all levels, so cross-scale information exchange happens inside the attention module itself — no FPN top-down path is needed (and the ablation confirms adding one does not help).

Two degenerate cases are worth noting: with $L=1$, $K=1$ and ${\mathbf{W}}_m^{\prime }$ fixed to identity, the module is deformable convolution; when the sampling points cover all locations, it is ordinary Transformer attention.

Complexity

Deformable attention costs $O(2N_q{C}^2+\min (HW{C}^2,N_{q}K{C}^2))$ (Appendix A.1; with $M=8$, $K\le 4$, $C=256$ the $5K+3MK<C$ so those terms vanish into the constant).

• In the encoder, $N_q=HW$, so complexity is $O(HW{C}^2)$ — linear in spatial size, versus DETR’s $O(H^2{W}^{2}C)$.
• In the decoder cross-attention, $N_q=N$ (number of object queries), so complexity is $O(NK{C}^2)$ — completely independent of $HW$.

Architecture

Encoder

Multi-scale feature maps $\{{\mathbf{x}}^l{\}}_{l=1}^L$ ($L=4$) are taken from ResNet stages $C_3$–$C_5$ via $1\times 1$ convolutions, plus a lowest-resolution level obtained by a $3\times 3$ stride-2 convolution on $C_5$. All levels have $C=256$ channels; strides are $8,16,32,64$.

All Transformer attention modules are replaced by multi-scale deformable attention. Query and key elements are both pixels of the multi-scale maps, and for each query pixel the reference point is the pixel itself. To let the model know which level a query came from, a learned scale-level embedding ${\mathbf{e}}_l$ is added to the feature alongside the (fixed) positional encoding; the $\{{\mathbf{e}}_l{\}}_{l=1}^L$ are randomly initialized and trained jointly.

Decoder

Only the cross-attention modules are replaced by multi-scale deformable attention — self-attention among object queries keeps standard Transformer attention, since its key elements are queries, not feature-map pixels.

For each object query, the 2-d normalized reference point ${\hat{\mathbf{p}}}_q\in [0,1{]}^2$ is predicted from the query embedding by a linear projection + sigmoid. Because deformable cross-attention gathers its features from around ${\hat{\mathbf{p}}}_q$, the detection head is made to predict the box relative to that same point rather than in absolute image coordinates (Appendix A.3):

${\hat{\mathbf{b}}}_q=\{\sigma (b_{qx}+{\sigma }^{-1}({\hat{p}}_{qx})),\sigma (b_{qy}+{\sigma }^{-1}({\hat{p}}_{qy})),\sigma (b_{qw}),\sigma (b_{qh})\}$

Reading it piece by piece. The head outputs four raw numbers $b_{q\{x,y,w,h\}}\in \mathbb{R}$. The width and height terms are the simple case: $\sigma (b_{qw})$ just squashes a raw number into $[0,1]$, since the box size is expressed as a fraction of the image and has nothing to offset against.

The center is where the reference point enters. We want “center = reference point, nudged a bit”, but ${\hat{p}}_{qx}$ already lives in the bounded $[0,1]$ interval, so adding a raw offset to it could leave that interval. The fix is to do the addition in logit space:

1. ${\sigma }^{-1}({\hat{p}}_{qx})$ maps the reference coordinate back to an unbounded real number (${\sigma }^{-1}(p)=\log \frac{p}{1-p}$).
2. Add the predicted offset $b_{qx}$ there, where addition is unconstrained and well-behaved.
3. $\sigma (\cdot )$ maps the sum back into $[0,1]$, guaranteeing ${\hat{\mathbf{b}}}_q\in [0,1{]}^4$ — valid normalized coordinates by construction, no clipping needed.

So ${\hat{\mathbf{p}}}_q$ is an initial guess of the box center and the head only has to learn a correction to it. Two consequences: the regression problem is easier (predict a small delta, not an absolute position), and the learned decoder attention ends up strongly correlated with the predicted box, since the points it samples and the box it outputs are anchored to the same ${\hat{\mathbf{p}}}_q$. Both accelerate convergence.

The same $\sigma (\Delta +{\sigma }^{-1}(\cdot ))$ trick reappears in the two variants below, where the thing being corrected is the previous decoder layer’s box instead of the reference point.

Variants

Both variants are enabled by the cheap, fast-converging backbone and can be stacked.

Iterative bounding box refinement. Each decoder layer refines the box predicted by the previous layer (inspired by RAFT’s iterative refinement). At layer $d$:

${\hat{\mathbf{b}}}_q^d=\{\sigma (\Delta b_{q\{x,y,w,h\}}^d+{\sigma }^{-1}({\hat{b}}_{q\{x,y,w,h\}}^{d-1}))\}$

Prediction heads are not shared across decoder layers, and gradients are blocked at ${\sigma }^{-1}({\hat{b}}_q^{d-1})$ to stabilize training. Initial boxes: ${\hat{b}}_{qx}^0={\hat{p}}_{qx}$, ${\hat{b}}_{qy}^0={\hat{p}}_{qy}$, ${\hat{b}}_{qw}^0={\hat{b}}_{qh}^0=0.1$ (results are robust to $0.05/0.1/0.2/0.5$). The cross-attention of layer $d$ also uses $({\hat{b}}_{qx}^{d-1},{\hat{b}}_{qy}^{d-1})$ as its reference point and modulates the sampling offsets by box size, $(\Delta p_{mlqkx}{\hat{b}}_{qw}^{d-1},\Delta p_{mlqky}{\hat{b}}_{qh}^{d-1})$, so sampling locations scale with the current box.

Two-stage Deformable DETR. Object queries in plain DETR are image-independent. Here the first stage is an encoder-only Deformable DETR used as a region proposal network: every pixel of the multi-scale maps is an object query and directly predicts a box (3-layer FFN for regression, linear layer for foreground/background classification), trained with the same Hungarian loss. Top-scoring boxes are picked as proposals — no NMS. The decoder is removed in this stage to avoid the quadratic self-attention cost over $HW$ queries. In the second stage, proposals become the initial boxes and their positional encodings become the object queries, then iterative refinement runs as above. Pixel $i$ at level $l_i$ with normalized coordinates ${\hat{\mathbf{p}}}_i$ predicts

${\hat{\mathbf{b}}}_i=\{\sigma (\Delta b_{ix}+{\sigma }^{-1}({\hat{p}}_{ix})),\sigma (\Delta b_{iy}+{\sigma }^{-1}({\hat{p}}_{iy})),\sigma (\Delta b_{iw}+{\sigma }^{-1}(2^{l_i-1}s)),\sigma (\Delta b_{ih}+{\sigma }^{-1}(2^{l_i-1}s))\}$

with base object scale $s=0.05$.

Hyperparameters

Reported training configuration (Section 5, Appendix A.4):

Setting	Value
Dataset	COCO 2017 (train / val / test-dev)
Backbone	ImageNet-pretrained ResNet-50 (default)
Feature levels	$L=4$, extracted from $C_3$–$C_5$ (+ stride-2 conv on $C_5$), $C=256$
Attention heads	$M=8$
Sampling points	$K=4$ per head per level
Object queries	$N=300$ (DETR uses 100)
Classification loss	Focal loss, loss weight 2
Epochs	50, LR decayed $\times 0.1$ at epoch 40
Optimizer	Adam, base LR $2\times 10^{-4}$, ${\beta }_1=0.9$, ${\beta }_2=0.999$, weight decay $10^{-4}$
LR multiplier	$\times 0.1$ for the linear projections predicting reference points and sampling offsets
Decoder layers	$D=6$
Hardware	NVIDIA Tesla V100

Initialization. ${\mathbf{W}}_m^{\prime },{\mathbf{W}}_m$ random; weights of the linear projection predicting $A_{mlqk}$ and $\Delta {\mathbf{p}}_{mlqk}$ set to zero; biases set so that at initialization $A_{mlqk}=\frac{1}{LK}$ and the $M=8$ heads sample in the 8 compass directions, $\Delta {\mathbf{p}}_{1lqk}=(-k,-k)$, $(-k,0)$, $(-k,k)$, $(0,-k)$, $(0,k)$, $(k,-k)$, $(k,0)$, $(k,k)$ for $k\in \{1,\dots ,K\}$. With iterative refinement, these offset biases are further scaled by $\frac{1}{2K}$ so initial sampling points lie inside the previous layer’s box.

Results

COCO 2017 val (Table 1):

Method	Epochs	AP	AP${}_S$	AP${}_M$	AP${}_L$	Params	FLOPs	GPU hours	FPS
Faster R-CNN + FPN	109	42.0	26.6	45.4	53.4	42M	180G	380	26
DETR	500	42.0	20.5	45.8	61.1	41M	86G	2000	28
DETR-DC5	500	43.3	22.5	47.3	61.1	41M	187G	7000	12
DETR-DC5${}^{+}$	50	36.2	16.3	39.2	53.9	41M	187G	700	12
Deformable DETR	50	43.8	26.4	47.1	58.0	40M	173G	325	19
+ iterative box refinement	50	45.4	26.8	48.3	61.7	40M	173G	325	19
++ two-stage	50	46.2	28.8	49.2	61.7	40M	173G	340	19

${\text{AP}}_S$ jumps from 20.5 to 26.4–28.8, i.e. the small-object gap to Faster R-CNN closes and then reverses. Training cost drops from 2000–7000 GPU hours to ~325.

Ablations on deformable attention (Table 2, COCO val): single-scale $K=1$ (≈ deformable convolution) gives 39.7 AP; $K=4$ single-scale 42.3; multi-scale inputs + $K=4$ but no multi-scale attention 42.3 → multi-scale deformable attention 43.8. So multi-scale inputs are worth $+1.7$ AP ($+2.9$ AP${}_S$), larger $K$ another $+0.9$, and cross-level attention $+1.5$. Adding FPN (43.8) or BiFPN (43.9) on top changes nothing — the module already exchanges information across levels.

State of the art (Table 3, COCO test-dev): ResNet-101 → 48.7 AP; ResNeXt-101 → 49.0 AP; ResNeXt-101 + DCN → 50.1 AP; with test-time augmentation (horizontal flip + multi-scale testing) → 52.3 AP.

What the model looks at (Appendix A.5/A.6): gradient-norm visualizations show Deformable DETR attends to the left/right object boundaries when predicting $x$ and $w$, and to top/bottom for $y$ and $h$ — similar to DETR. Unlike DETR, it also looks at pixels inside the object, which is what the category score needs; decoder attention focuses on the whole foreground instance rather than only extreme points.

Limitations

• Despite FLOPs on par with Faster R-CNN + FPN, inference is ~25% slower (19 vs 26 FPS) because deformable sampling implies unordered memory access. It is still $1.6\times$ faster than DETR-DC5, whose bottleneck is memory access in dense attention.
• Sampling offsets and attention weights come from the query alone; the module never compares query and key content, so it is a pre-filter heuristic rather than full content-based retrieval.
• Performance on large objects (AP${}_L$ 58.0 without refinement) is below DETR’s 61.1 — the global receptive field of dense attention still helps there, and only the refinement variants recover it.
• Custom CUDA kernels are required for the multi-scale deformable attention op; it is not expressible efficiently with stock framework primitives.

See also

• DETR — the base architecture this fixes
• Conditional DETR — a lighter-touch answer to the same convergence problem, changing only the decoder query
• YOLOv10 — another NMS-free end-to-end detector, from the YOLO line
• Implementing Research Papers