DAB-DETR

DAB-DETR (Dynamic Anchor Boxes are Better Queries for DETR) is an object detector that uses explicit anchor boxes as decoder queries.

• Paper: Liu, Li, Zhang, Yang, Qi, Su, Zhu and Zhang (Tsinghua / IDEA / HKUST / Peng Cheng Lab), ICLR 2022, arXiv:2201.12329.
• Core idea: replaces DETR’s opaque learnable query vectors with explicit 4D anchor boxes $(x,y,w,h)$ that are fed through the decoder as positional queries and updated layer by layer.
• Consequence: because the query now carries a size as well as a position, the cross-attention positional prior can be modulated by the anchor’s width and height into an oval Gaussian matched to the object’s scale — which makes each decoder layer readable as a soft ROI pooling step in a cascade.
• Headline result: with a ResNet-50-DC5 backbone and a 50-epoch schedule it reaches 45.7 AP on COCO, the best among single-scale DETR-like models at the time.

Motivation: what is wrong with learnable queries

Each query in a DETR decoder splits into a content part (the decoder embedding) and a positional part (the learnable query); keys split the same way into an encoded image feature and a positional embedding. Cross-attention is therefore a similarity that mixes a semantic term and a positional term, and the positional term is supposed to constrain where in the feature map the query pools features from.

The paper isolates the failure by elimination. Sun et al.’s hypothesis is that cross-attention is responsible for slow convergence; DAB-DETR narrows this further. Encoder self-attention and decoder cross-attention have identical key/value components, so the difference lies entirely in the queries. Two candidate causes:

1. Optimization difficulty of learning the queries. Ruled out empirically: reusing well-trained DETR queries frozen, and training only everything else, improves convergence only marginally over the first ~25 epochs (Fig. 3a).
2. Bad positional priors. This is the culprit. Visualizing the positional attention map $\text{PE}(\text{query})\cdot \text{PE}(\text{key})$ shows DETR’s learned queries produce multi-modal, nearly uniform attention maps — several concentration centers per query, or blobs that are far too large or too small. A query with two modes cannot pick out a single object when several are present, and a diffuse one injects no useful positional information at all.

Replacing DETR’s query formulation with dynamic anchor boxes — changing nothing else, no 300 queries, no focal loss — already gives both faster convergence and higher AP (DETR+DAB in Fig. 3b), which is the paper’s evidence that the multi-mode property is the root cause.

Conditional DETR fixes the multi-modality by generating positional queries in the same sinusoidal space as the image positional embeddings, producing clean Gaussian-like maps — but an isotropic, fixed-size Gaussian, wrong for objects of different aspect ratios and scales. Deformable DETR and Anchor DETR likewise carry only 2D reference points. DAB-DETR’s claim is that all information a query needs to hold is box coordinates, and that the missing $(w,h)$ is what lets the prior adapt.

Architecture

The backbone, encoder and Hungarian set-prediction loss are DETR’s. Everything changes in the decoder, which now consumes dual queries: content queries $C_q\in {\mathbb{R}}^D$ (decoder embeddings) and positional queries $P_q\in {\mathbb{R}}^D$ derived from an anchor $A_q=(x_q,y_q,w_q,h_q)$, with $N=300$ anchors as learnable parameters.

Anchors as positional queries

The positional query is produced from the anchor by sinusoidal encoding followed by a shared MLP:

$P_q=\text{MLP}(\text{PE}(A_q))$

with the encoding overloaded to the 4-tuple by concatenation,

$\text{PE}(A_q)=\text{Cat}(\text{PE}(x_q),\text{PE}(y_q),\text{PE}(w_q),\text{PE}(h_q))$

Each scalar maps to $D/2$ dimensions, so $\text{PE}:{\mathbb{R}}^4\to {\mathbb{R}}^{2D}$ and $\text{MLP}:{\mathbb{R}}^{2D}\to {\mathbb{R}}^D$ reduces back. The MLP has two linear+ReLU submodules with the reduction at the first linear layer, and its parameters are shared across all decoder layers.

Self-attention adds position to content in the usual way,

$Q_q=C_q+P_q,K_q=C_q+P_q,V_q=C_q$

Cross-attention instead concatenates the parts, as in Conditional DETR, so that the query-to-feature similarity decomposes cleanly into a content term and a positional term:

$Q_q=\text{Cat}(C_q,\text{PE}(x_q,y_q)\cdot {\text{MLP}}^{\text{(csq)}}(C_q)),K_{x,y}=\text{Cat}(F_{x,y},\text{PE}(x,y)),V_{x,y}=F_{x,y}$

where $F_{x,y}\in {\mathbb{R}}^D$ is the image feature at position $(x,y)$ and $\cdot$ is element-wise multiplication. ${\text{MLP}}^{\text{(csq)}}:{\mathbb{R}}^D\to {\mathbb{R}}^D$ produces a content-conditioned scale vector — the conditional spatial query mechanism, reused here to rescale the positional embedding. Note that only the anchor center $(x_q,y_q)$ enters the query embedding; the size $(w_q,h_q)$ enters through the modulation described below. Both query and key positional embeddings are generated from 2D coordinates in the same way, so the positional similarity is comparable by construction.

Anchor update

Because the query is a box, it can be refined in coordinate space — which is impossible when the query is a high-dimensional vector, as in DETR and Conditional DETR, since there is no way to map an updated box back to an embedding. Each decoder layer’s prediction head emits relative offsets $(\Delta x,\Delta y,\Delta w,\Delta h)$ and the anchor becomes

$(x,y,w,h)\leftarrow (x+\Delta x,y+\Delta y,w+\Delta w,h+\Delta h)$

feeding the next layer. All prediction heads across layers share the same parameters. This is the same idea as Deformable DETR’s iterative bounding box refinement, but decoupled from deformable attention and therefore applicable to a plain dense-attention decoder. Since the refined anchor regenerates both the self-attention and the cross-attention positional queries of the next layer, every layer benefits from the update — unlike Anchor DETR and Deformable DETR, whose positional queries are passed unadapted to self-attention (Appendix B, Fig. 8).

The layer-by-layer picture is thus a cascade of soft ROI pooling operations: each layer pools features from the region its current anchor defines, and hands a corrected anchor to the next.

Width & height-modulated Gaussian kernel

The unmodulated positional attention logit is the sum of the two coordinate dot products,

$\text{Attn}((x,y),(x_{\text{ref}},y_{\text{ref}}))=(\text{PE}(x)\cdot \text{PE}(x_{\text{ref}})+\text{PE}(y)\cdot \text{PE}(y_{\text{ref}}))/\sqrt{D}$

which is isotropic: the same circular Gaussian blob regardless of the object. DAB-DETR divides the $x$ and $y$ halves separately by the anchor’s relative width and height, before the softmax:

$\text{ModulateAttn}((x,y),(x_{\text{ref}},y_{\text{ref}}))=(\text{PE}(x)\cdot \text{PE}(x_{\text{ref}})\frac{w_{q,\text{ref}}}{w_q}+\text{PE}(y)\cdot \text{PE}(y_{\text{ref}})\frac{h_{q,\text{ref}}}{h_q})/\sqrt{D}$

$w_q,h_q$ are the anchor’s own width and height, and the reference sizes are predicted from the content query,

$w_{q,\text{ref}},h_{q,\text{ref}}=\sigma (\text{MLP}(C_q))$

A large $w_q$ shrinks the $x$ term, flattening the attention along $x$ and widening the pooled band; the two ratios independently stretch the Gaussian into an oval matched to the anchor’s aspect ratio (Fig. 6, left of the figure above: $H=1,W=3$ gives a horizontally elongated map, $H=3,W=1$ a vertical one).

Temperature tuning

The sinusoidal encoding is

$\text{PE}(x{)}_{2i}=\sin (\frac{x}{T^{2i/D}}),\text{PE}(x{)}_{2i+1}=\cos \left(\frac{x}{T^{2i/D}}\right)$

with $T$ a hand-set temperature that controls how flat the resulting positional prior is (right of the figure above). This is a genuinely overlooked hyperparameter: the canonical $T=10000$ comes from language modelling, where $x$ is an integer token index, whereas here $x$ is a normalized box coordinate in $[0,1]$ — a completely different input range. The paper empirically picks $T=20$ for all models.

Hyperparameters

Reported in Appendix A (Training Details):

Setting	Value
Dataset	COCO 2017 (train2017 / val2017)
Backbones	ImageNet-pretrained ResNet-50, ResNet-101 and their $16\times$-resolution DC5 variants
Encoder / decoder layers	6 / 6
Anchors (queries)	$N=300$
Pattern embeddings	3 (in the ${}^{\ast }$ variants, as in Anchor DETR)
Activation	PReLU
Prediction heads	Box and label MLPs share parameters across decoder layers
Positional-encoding temperature	$T=20$
Classification loss	Focal loss, $\alpha =0.25$, $\gamma =2$; weight 2.0 in matching, 1.0 in the final loss
Box losses	L1 weight 5.0, GIoU weight 2.0 (identical in matching and final loss)
Optimizer	AdamW, weight decay $10^{-4}$
Learning rate	$10^{-5}$ backbone, $10^{-4}$ everything else
Schedule	50 epochs, LR $\times 0.1$ at epoch 40
Batch size	64 for the hyperparameter search, 16 for all reported results
Hardware	16 GPUs, 1 image per GPU; NVIDIA A100

GPU memory ranges from 3.6 GB (DAB-DETR-R50${}^{\ast }$, batch 1/GPU) to 16.7 GB (DAB-DETR-R101-DC5${}^{\ast }$) — Table 4.

Results

COCO 2017 val, all at 50 epochs unless noted; ${}^{\ast }$ marks the 3-pattern-embedding variants (Table 2):

Model	Multi-scale	Epochs	AP	AP${}_S$	AP${}_M$	AP${}_L$	GFLOPs	Params
DETR-R50		500	42.0	20.5	45.8	61.1	86	41M
Conditional DETR-R50		50	40.9	20.8	44.6	59.2	90	44M
DAB-DETR-R50		50	42.2	21.5	45.7	60.3	94	44M
DAB-DETR-R50${}^{\ast }$		50	42.6	21.8	46.2	61.1	100	44M
DETR-DC5-R50		500	43.3	22.5	47.3	61.1	187	41M
Deformable DETR-R50	✓	50	43.8	26.4	47.1	58.0	173	40M
SMCA-R50	✓	50	43.7	24.2	47.0	60.4	152	40M
Conditional DETR-DC5-R50		50	43.8	24.0	47.6	60.7	195	44M
DAB-DETR-DC5-R50		50	44.5	25.3	48.2	62.3	202	44M
DAB-DETR-DC5-R50${}^{\ast }$		50	45.7	26.1	49.4	63.1	216	44M
DETR-DC5-R101		500	44.9	23.7	49.5	62.3	253	60M
Conditional DETR-DC5-R101		50	45.0	26.1	48.9	62.8	262	63M
DAB-DETR-DC5-R101${}^{\ast }$		50	46.6	28.1	50.5	64.1	296	63M

The DC5-R50${}^{\ast }$ result (45.7) beats the multi-scale Deformable DETR and SMCA at the same epoch budget, on a single-scale feature map.

Convergence is faster than Conditional DETR’s throughout, and most visibly in the early epochs (Fig. 13). Runtime is essentially the same as Conditional DETR’s — 0.059 vs 0.057 s/img for R50, 0.085 vs 0.082 for R101 on an A100 (Table 9) — so the extra AP is free at inference.

Ablations

Component ablation (Table 3, R50-DC5): full model 45.7 AP; without anchor update 44.0; without $w/h$-modulated attention 45.0; 2D anchor points instead of 4D boxes 44.0; without temperature tuning 44.4. So the 4D box formulation is worth $+1.0$ AP over 2D points, the layer-by-layer anchor update $+1.7$, modulated attention $+0.7$ and temperature tuning $+1.3$.

Temperature (Table 6, R50, no patterns, no modulation): $T=2$ → 39.6, $T=5$ → 40.0, $T=10$ → 40.0, $T=20$ → 40.1, $T=50$ → 39.8, $T=100$ → 39.8, $T=10000$ → 39.5. Larger $T$ flattens the prior and helps large objects; $T=2$ gives the best AP${}_S$/AP${}_M$ (19.3/43.3 vs 18.9/42.6 at $T=10000$) while $T=10000$ gives the best AP${}_L$ — the trade-off is exactly what a positional prior should show.

Decoder depth (Table 7, R50-DC5): 2 layers → 40.2 AP, 3 → 43.9, 4 → 44.9, 5 → 45.2, 6 → 45.7. The cascade of soft ROI pooling saturates slowly.

Generality: DAB-Deformable-DETR (Appendix C, Table 5). Reformulating Deformable DETR’s queries as dynamic anchor boxes — “no more than 10 lines of code” — lifts the open-source Deformable DETR+ from 46.3 to 46.8 AP ($+0.5$) under an identical R50 multi-scale setting, with AP${}_S$ 28.6 → 29.1. Interestingly the total loss summed over decoder layers is higher for DAB-Deformable-DETR while the last layer’s loss is lower — and only the last layer is used at inference.

Fixing $x,y$ (Appendix G, Table 8). Freezing the first-layer anchor centers at their random initialization (they are still updated in later layers) improves AP consistently: DAB-DETR-R50${}^{\ast }$ 42.6 → 42.9, R101${}^{\ast }$ 44.1 → 44.8, DC5-R101${}^{\ast }$ 46.6 → 46.7. The authors conjecture that the randomly initialized fixed centers act as a regularizer against overfitting. Figure 10 shows why the learned centers behave oddly in the first place: anchor points learned alone spread evenly over the image, whereas the centers of learned anchor boxes look randomly scattered, apparently because center and size are coupled.

Limitations

• Single-scale. The reported models use one feature map (high resolution bought with the expensive DC5 backbone). Failure cases (Fig. 12) are dense scenes, very small objects and very large objects; the paper explicitly names introducing a multi-scale technique as future work, and DAB-Deformable-DETR is the stopgap.
• Temperature is hand-tuned. $T=20$ is chosen empirically and is a single global constant, even though the ablation shows the optimum differs per object scale — a per-scale or learned temperature is left open.
• A first-layer anchor initialization the model cannot learn well. That randomly initialized fixed centers beat learned ones means the learned initial anchor distribution is actively suboptimal, and the paper offers only a conjecture as to why.
• Matching instability is untouched. DAB-DETR sharpens the positional prior but leaves the bipartite matching itself unstable early in training — a query can be matched to different ground-truth objects across epochs. That is exactly what DN-DETR adds on top of this architecture (denoising training with noised ground-truth boxes as extra queries), and DINO then combines dynamic anchor boxes, denoising with negative samples, and a mixed query selection scheme.

See also

• DETR — the base architecture whose queries this reinterprets
• Conditional DETR — the concatenated content/position cross-attention and conditional spatial query that DAB-DETR builds on
• Deformable DETR — source of iterative box refinement; DAB-Deformable-DETR is the combination
• DN-DETR — adds denoising training directly on top of DAB-DETR’s anchor queries
• DINO — the successor combining dynamic anchor boxes, contrastive denoising and query selection
• Implementing Research Papers