DN-DETR

DN-DETR is a training method, not an architecture: it adds a query denoising auxiliary task to any DETR-like detector to make it converge faster.

• Paper: Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni and Lei Zhang (HKUST / Tsinghua / IDEA), DN-DETR: Accelerate DETR Training by Introducing Query DeNoising, CVPR 2022 oral; extended journal version arXiv:2203.01305.
• Diagnosis: differs from that of Deformable DETR and Conditional DETR, which both blame the decoder’s cross-attention. DN-DETR blames the bipartite (Hungarian) matching itself, arguing that the assignment of ground-truth objects to queries flips from epoch to epoch, giving each query an inconsistent optimization target early in training.
• Fix: feed noised ground-truth boxes and labels as extra decoder queries alongside the ordinary learnable ones, and train the decoder to reconstruct the originals — a task with a fixed, unambiguous target that bypasses matching entirely.
• Headline result: bolted onto DAB-DETR it gives $+1.9$ AP at 50 epochs (44.1 vs 42.2, ResNet-50) and $+3.7$ AP in the 12-epoch $1\times$ schedule, at essentially zero inference cost.

Motivation: bipartite matching is unstable

DETR turns ground-truth assignment into a dynamic process — which query is responsible for which object is re-decided every forward pass by the Hungarian algorithm on the current cost matrix. A small change in the cost matrix can flip the whole assignment (the stable-matching literature notes that Hungarian matching is not stable when blocking pairs exist), so the same query is matched to different objects in different epochs. The paper frames DETR-like training as two entangled stages — learning good anchors and learning relative offsets from those anchors — and observes that inconsistent anchor updates make offset learning much harder.

To make this quantitative the paper defines an instability metric $IS$. For a training image, let ${\mathbf{O}}^i=\{O_0^i,\dots ,O_{N-1}^i\}$ be the $N$ predicted objects in epoch $i$ and $\mathbf{T}=\{T_0,\dots ,T_{M-1}\}$ the $M$ ground-truth objects. After bipartite matching, store the assignment in an index vector ${\mathbf{V}}^i=\{V_0^i,\dots ,V_{N-1}^i\}$:

$V_n^i=\{\begin{array}{ll}m,& \text{if }{O}_n^i\text{ matches }{T}_m\\ -1,& \text{if }{O}_n^i\text{ matches nothing}\end{array}$

The instability of epoch $i$ is the number of queries whose assignment changed since the previous epoch:

$I{S}^i={\sum }_{n=0}^{N}1(V_n^i\ne V_n^{i-1})$

averaged over the dataset. COCO 2017 val has 7.36 objects per image on average, so the largest possible $IS$ is $7.36\times 2=14.72$ (every old match broken and a new one made).

Measured over 12 epochs, DETR and DAB-DETR sit around $IS\approx 8$–$11$ with a conspicuous rebound at epochs 8–9 (the learning-rate drop reshuffles the matching); DN-DETR decreases monotonically to $\approx 5.7$. A second measurement (Fig. 3 in the paper) tracks the mean ${\ell }_1$ distance between the last decoder layer’s initial anchors and their matched GT boxes: DAB-DETR oscillates between 0.10 and 0.15, DN-DETR falls smoothly from 0.11 to 0.08. Denoising both stabilizes the assignment and makes each query search more locally.

Architecture

DN-DETR is built on DAB-DETR, where each decoder query is explicitly a 4D anchor box $(x,y,w,h)$ that is dynamically updated layer by layer (each decoder layer emits $(\Delta x,\Delta y,\Delta w,\Delta h)$ and the anchor becomes $(x+\Delta x,y+\Delta y,w+\Delta w,h+\Delta h)$). The 4D formulation is what makes denoising natural: a noised GT box has exactly the same format as a query, so it can be dropped straight into the decoder.

Only two architectural modifications are made to DAB-DETR:

1. The generic decoder embedding (content part of the query) is replaced by a class label embedding, so that labels can be noised and reconstructed too.
2. An indicator is appended to the label embedding: $1$ if the query belongs to the denoising part, $0$ if it belongs to the matching part.

Writing $D$ for the Transformer decoder, $F$ for the refined image features from the encoder and $A$ for the attention mask, plain DETR-like inference is $\mathbf{o}=D(\mathbf{q},F\mid A)$, and DN-DETR’s training forward pass is

$\mathbf{o}=D(\mathbf{q},\mathbf{Q},F\mid A)$

with $\mathbf{q}=\{q_0,\dots ,q_{K-1}\}$ the denoising part (noised GT objects, supervised by a reconstruction loss) and $\mathbf{Q}=\{Q_0,\dots ,Q_{L-1}\}$ the matching part (ordinary learnable anchors, supervised by the usual Hungarian loss). At inference the denoising part is removed entirely — DN-DETR is exactly DAB-DETR at test time, with identical FLOPs and parameters.

Denoising groups

To use the auxiliary task more efficiently the same GT set is noised several times. The denoising part becomes

$\mathbf{q}=\{{\mathbf{g}}_0,{\mathbf{g}}_1,\dots ,{\mathbf{g}}_{P-1}\},{\mathbf{g}}_p=\{q_0^p,q_1^p,\dots ,q_{M-1}^p\},q_m^p=\delta (t_m)$

where ${\mathbf{g}}_p$ is the $p$-th denoising group, $M$ is the number of GT objects in the image, $t_m$ the $m$-th GT object and $\delta (\cdot )$ the noising function. Each group is an independently noised copy of all GT objects, so with $P$ groups the decoder processes $P\times M+N$ queries during training. The cost is negligible: with 5 groups, training GFLOPs for DAB-DETR-R50 go from 94.4 to 94.6, and testing is unaffected.

Noise

Boxes are noised in two ways, controlled by ${\lambda }_1,{\lambda }_2\in (0,1)$:

• Center shifting. A random offset $(\Delta x,\Delta y)$ with $|\Delta x|<\frac{{\lambda }_{1}w}{2}$ and $|\Delta y|<\frac{{\lambda }_{1}h}{2}$, so the noised center still lies inside the original box.
• Box scaling. The width and height are resampled uniformly from $[(1-{\lambda }_2)w,(1+{\lambda }_2)w]$ and $[(1-{\lambda }_2)h,(1+{\lambda }_2)h]$.

Labels are noised by label flipping: a fraction $\gamma$ of GT labels are randomly replaced by another class. Because the box is (mostly) correct while the label is wrong, this forces the model to predict the label from the box content rather than from the query’s own prior, which is what makes it capture the label–box relationship.

The reconstruction loss is ${\ell }_1$ + GIoU for boxes and focal loss for class labels, as in DAB-DETR. Since a noised query is a “good anchor” with a corresponding GT box nearby, the denoising task has a clear, fixed target — learn the offset from a nearby anchor — with none of the ambiguity of Hungarian matching.

Label embedding and the unknown class

Beyond COCO’s 80 classes an extra unknown class embedding is used in the matching part, so that matching-part queries are semantically consistent with denoising-part ones (they carry a label embedding too, just an uninformative one). The indicator bit is appended to distinguish the two parts.

Attention mask

The attention mask is the component that makes the whole scheme work rather than cheat. There are two leakage paths to close:

1. The matching part must not see the denoising part, or it would read off the noised GT objects and trivially predict them.
2. One denoising group must not see another, since different groups are noised versions of the same GT objects.

With $\mathbf{A}=[a_{ij}{]}_{W\times W}$, $W=P\times M+N$, where the first $P\times M$ rows/columns are the denoising part and $a_{ij}=1$ means query $i$ cannot see query $j$:

$a_{ij}=\{\begin{array}{ll}1,& \text{if }j<P\times M\text{ and }\lfloor \frac{i}{M}\rfloor \ne \lfloor \frac{j}{M}\rfloor \\ 1,& \text{if }j<P\times M\text{ and }i\ge P\times M\\ 0,& \text{otherwise}\end{array}$

The converse direction does not matter: whether the denoising part can see the matching part is irrelevant, because matching-part queries are learned parameters carrying no information about the GT objects of this particular image. The ablation is stark — without the mask, denoising training drops the model to 24.0 AP (vs 43.4 with it), i.e. it actively destroys the detector.

Applying DN to other detectors

Denoising is a training method, so it transfers to any model whose queries (or proposals) can be interpreted positionally:

• DN-Deformable-DETR — Deformable DETR with its queries reformulated as 4D anchor boxes as in DAB-DETR, plus denoising. DN-Deformable-DETR++ additionally puts deformable attention in the decoder as well as the encoder.
• DN-Anchor-DETR — Anchor DETR uses 2D anchor points, so only the center-shifting noise applies.
• DN-Vanilla-DETR — DETR’s positional queries are high-dimensional vectors with no explicit geometric meaning; a linear box embedding maps noised 4D boxes into the same latent space, and the label embedding supplies the content query.
• DN-Faster R-CNN — noised GT boxes are fed to the detection head in parallel with the RPN’s proposals. Faster R-CNN has no bipartite matching, but it does have IoU-threshold-based label assignment, and denoising works as a shortcut to box regression without label assignment.
• DN-Mask2Former / Mask DINO — for segmentation, noised GT masks (shifted on both axes) are fed to the decoder as attention masks, with the objective of reconstructing the original mask.

The method is also a direct ancestor of DINO, which extends it to Contrastive DeNoising (CDN) by adding hard negative samples (boxes noised beyond a larger threshold, trained to predict “no object”) so the model learns to reject as well as to reconstruct.

Hyperparameters

Reported training configuration (Section 5.1):

Setting	Value
Dataset	MS-COCO 2017 (118K train / 5K val / 41K test-dev)
Backbones	ResNet-50, ResNet-101, and their $16\times$-resolution DC5 variants (ImageNet-pretrained)
Encoder / decoder	6 layers each, hidden dimension 256
Queries (matching part)	$N=300$
Center-shift noise	${\lambda }_1=0.4$
Box-scale noise	${\lambda }_2=0.4$
Label-flip ratio	$\gamma =0.2$
Denoising groups	5 by default (10 for DN-Deformable-DETR and DN-Faster R-CNN, 5 for DN-Deformable-DETR++, DINO, Anchor DETR, Mask2Former)
Optimizer	AdamW, weight decay $10^{-4}$
Learning rate	$10^{-4}$, $\times 0.1$ at epoch 40 (50-epoch schedule) or epoch 11 (12-epoch $1\times$ schedule)
Batch size	16
Hardware	8 $\times$ NVIDIA A100
Reconstruction loss	${\ell }_1$ + GIoU (boxes), focal loss (labels)
Pattern embeddings	3, for the Anchor DETR comparisons

Noise is drawn from a uniform distribution in all cases.

Results

50-epoch single-scale setting (Table 1). Gains over the DAB-DETR baseline under identical settings:

Model	Epochs	AP	AP${}_{50}$	AP${}_S$	AP${}_M$	AP${}_L$	GFLOPs	Params
DETR-R50	500	42.0	62.4	20.5	45.8	61.1	86	41M
Conditional DETR-R50	50	40.9	61.8	20.8	44.6	59.2	90	44M
DAB-DETR-R50	50	42.2	63.1	21.5	45.7	60.3	94	44M
DN-DETR-R50	50	44.1 (+1.9)	64.4	22.9	48.0	63.4	94	44M
DAB-DETR-R101	50	43.5	63.9	23.6	47.3	61.5	174	63M
DN-DETR-R101	50	45.2 (+1.7)	65.5	24.1	49.1	65.1	174	63M
DAB-DETR-DC5-R50	50	44.5	65.1	25.3	48.2	62.3	202	44M
DN-DETR-DC5-R50	50	46.3 (+1.8)	66.4	26.7	50.0	64.3	202	44M
DAB-DETR-DC5-R101	50	45.8	65.9	27.0	49.8	63.8	282	63M
DN-DETR-DC5-R101	50	47.3 (+1.5)	67.5	28.6	51.5	65.0	282	63M

Parameters and GFLOPs are unchanged in every row — the gain is free at test time.

12-epoch ($1\times$) setting (Table 2), where the instability argument bites hardest. DETR-R50 manages only 15.5 AP in 12 epochs; DN-DETR-DC5-R50 reaches 41.7 vs DAB-DETR-DC5-R50’s 38.0 ($+3.7$), and DN-DETR-DC5-R101 42.8 vs 40.3 ($+2.5$). With multi-scale deformable attention, DN-Deformable-DETR-R50 gets 43.4 and DN-Deformable-DETR-R50++ 46.0 in 12 epochs. Most striking: DN-Deformable-DETR-R101 achieves 44.1 AP in 12 epochs, matching Faster R-CNN-FPN-R101’s 44.0 at 108 epochs — a $9\times$ speedup.

Best multi-scale results at 50 epochs (Table 4, ResNet-50): Deformable DETR 43.8, SMCA 43.7, Dynamic DETR 47.2, DAB-Deformable-DETR 46.9, DN-Deformable-DETR 48.6, DN-Deformable-DETR++ 49.5. The $+1.7$ over the strong DAB-Deformable-DETR baseline isolates the denoising contribution from the anchor-box query formulation.

$2\times$ acceleration. DN models at half the epochs match the baselines at full length: DN-DETR-DC5-R50 at 25 epochs → 44.4 vs DAB-DETR-DC5-R50 at 50 → 44.5; DN-Deformable-DETR-R50 at 25 → 46.8 vs 46.9; DN-Vanilla-DETR at 250 epochs → 42.2 vs DETR at 500 → 42.0. In wall-clock terms on 8 A100s, DAB-DETR-R50 takes 2555 min (51.1 min/epoch $\times$ 50) and DN-DAB-DETR-R50 1443 min (57.7 min/epoch $\times$ 25) for the same AP — a ~13% per-epoch overhead bought back many times over.

Transfer to other models (Table 3, 12 epochs unless noted): DN-Anchor-DETR-DC5-R50 39.4 ($+1.2$), DN-Group-DAB-DETR-DC5-R50 44.5 ($+2.6$), DN-Faster R-CNN-FPN-R50 38.4 ($+0.5$), DN-SAM-DETR++-R50 44.8 ($+1.6$), DINO-R50 with DN 47.4 ($+1.4$), DN-Vanilla-DETR-R50 42.6 ($+2.0$) at 300 epochs, Mask DINO-R50 with mask DN 41.4 ($+0.7$), DN-Mask2Former-R50 39.7 ($+1.0$). Denoising is therefore compatible with 2D anchor points, 4D anchor boxes, high-dimensional query vectors, deformable attention, semantic alignment, query selection, and mask heads alike.

Ablations

Components (Table 5, ResNet-50, 50 epochs, 1 denoising group):

Box denoising	Label denoising	Attention mask	AP
✓	✓	✓	43.4
✓		✓	43.0
		✓	42.2
✓	✓		24.0

Box denoising is worth $+0.8$, label denoising a further $+0.4$, and omitting the attention mask is catastrophic.

Number of denoising groups (Table 6): more groups help, with diminishing returns.

Backbone	No group	1 group	5 groups
R50	42.2	43.4	44.1
R50-DC5	44.5	45.6	46.3
R101	43.5	45.0	45.2
R101-DC5	45.8	46.5	47.3

Noise scale (Fig. 8; 20 epochs, batch size 64, ResNet-50, no LR drop — one scale fixed at 0.4 while the other varies):

Both noise types help, but the curves are unimodal: the center-shift scale peaks at ${\lambda }_1=0.2$ (35.5 AP) and the box-scale noise at ${\lambda }_2=0.4$ (35.7 AP), and by 0.8 both have fallen back to ~33.9, at or below the no-noise value. Too much noise turns “reconstruct a nearby box” into a task with no signal.

Convergence curves (Fig. 7): with DC5-R50, DN-DETR reaches 40 AP in 20 epochs where DAB-DETR needs 40, and finishes at 46.3 vs 44.3. In the multi-scale setting DN-Deformable-DETR reaches 47.8 AP in 30 epochs — 0.9 above the converged DAB-Deformable-DETR.

Known-label detection

Because queries are decomposed into an anchor box and a class label embedding, the model can be given the set of class labels present in an image (without boxes) and asked to detect them. Concatenating each known class embedding with indicator $1$ and feeding it to the decoder raises AP from 43.4 (no known labels) to 43.8 after 1 epoch of finetuning and 46.6 after 10 — a preliminary result the paper offers as evidence that classification information, if available, substantially eases detection, and as a route toward zero-shot / open-set detection using pretrained language-model class embeddings.

Limitations

• The noise is drawn from a plain uniform distribution; the paper explicitly leaves more sophisticated noising schemes (as used in diffusion models) to future work.
• Denoising supplies only positive anchors — every noised query has a GT box to reconstruct — so the model never learns to reject a nearby-but-wrong anchor. This is precisely the gap DINO closes with contrastive denoising.
• The auxiliary task adds ~13% wall-clock per epoch and, with many groups, extra decoder self-attention over $P\times M+N$ queries; the number of denoising queries scales with the number of GT objects, so crowded images cost more.
• The correctness of the whole scheme rests on the attention mask; getting it wrong does not degrade gracefully but collapses performance (43.4 → 24.0 AP).
• The gains are largest in short schedules; at 50 epochs on single-scale backbones the improvement is $+1.5$ to $+1.9$ AP, useful but not transformative, and DN-DETR does nothing about the encoder cost or the multi-scale feature problem that Deformable DETR targets.
• Known-object / known-label detection is presented as a preliminary exploration on models that were not hyperparameter-tuned for it.

See also

• DAB-DETR — the 4D-anchor-box query formulation DN-DETR is built on
• DETR — the original set-prediction detector whose Hungarian matching this analyzes
• Deformable DETR — the multi-scale sparse-attention answer to slow convergence, combined with DN in DN-Deformable-DETR
• Conditional DETR — the decoder-query answer to the same problem
• DINO — the successor, which turns denoising into contrastive denoising
• Task-Aligned-Assignment — a different take on the label-assignment instability problem
• Implementing Research Papers