Conditional DETR

Conditional DETR is an object detector that speeds up DETR training by changing only how the decoder cross-attention query is formed.

• Paper: Meng, Chen, Fan, Zeng, Li, Yuan, Sun and Wang (USTC / Peking University / Microsoft Research Asia), Conditional DETR for Fast Training Convergence, ICCV 2021, arXiv:2108.06152.
• Problem: the same as Deformable DETR — DETR’s 500-epoch training schedule — but with a much smaller intervention.
• Core idea: keeps global dense attention everywhere and changes only how the decoder cross-attention query is formed. Each query is given a conditional spatial query, predicted from the previous decoder layer’s embedding and a reference point, which lets every cross-attention head attend to a narrow spatial band around one object extremity.
• Headline result: training converges $6.7\times$ faster for R50/R101 backbones and $10\times$ faster for the high-resolution DC5 variants.

Motivation: the content query is overloaded

In DETR’s decoder cross-attention, the key is ${\mathbf{c}}_k+{\mathbf{p}}_k$ (content embedding from the encoder plus positional embedding of its 2D location) and the query is ${\mathbf{c}}_q+{\mathbf{p}}_q$ (decoder self-attention output plus object query ${\mathbf{o}}_q$). The attention logit expands into four terms:

$({\mathbf{c}}_q+{\mathbf{p}}_q{)}^{\top }({\mathbf{c}}_k+{\mathbf{p}}_k)={\mathbf{c}}_q^{\top }{\mathbf{c}}_k+{\mathbf{c}}_q^{\top }{\mathbf{p}}_k+{\mathbf{o}}_q^{\top }{\mathbf{c}}_k+{\mathbf{o}}_q^{\top }{\mathbf{p}}_k$

The paper’s empirical starting point: removing the positional terms from the second decoder layer onward costs only ~1.4 AP (34.9 → 34.0 at 50 epochs). So the spatial query ${\mathbf{o}}_q$ contributes little and the whole burden of localization falls on the content query ${\mathbf{c}}_q$, which must simultaneously

1. match the content key ${\mathbf{c}}_k$ (find the object), and
2. match the spatial key ${\mathbf{p}}_k$ (find where in the object — the four box extremities needed for regression).

The object query gives only a generic, image-independent attention pattern; it cannot shrink the spatial search range for a specific instance. A content embedding strong enough to serve both roles is exactly what takes hundreds of epochs to learn. Figure 1 shows the consequence: DETR at 50 epochs (second row) fails to highlight the left and right edges at all, and only at 500 epochs (third row) do all four extremity bands appear.

Conditional cross-attention

The fix is to stop adding the spatial and content parts and instead concatenate them, so that the dot product splits cleanly into two independent halves:

${\mathbf{c}}_q^{\top }{\mathbf{c}}_k+{\mathbf{p}}_q^{\top }{\mathbf{p}}_k$

The query is $[{\mathbf{c}}_q;{\mathbf{p}}_q]$ and the key is $[{\mathbf{c}}_k;{\mathbf{p}}_k]$. The content dot product handles what (appearance matching for classification), the spatial dot product handles where (localization). Neither role contaminates the other, and ${\mathbf{c}}_q$ no longer needs to encode positional information.

The whole novelty is then in constructing ${\mathbf{p}}_q$ — a query that is conditional on the image, unlike DETR’s fixed learned ${\mathbf{o}}_q$.

Where the spatial information comes from

DETR predicts a box from the decoder embedding $\mathbf{f}$ relative to a reference point $\mathbf{s}$:

$\mathbf{b}=\mathrm{sigmoid}(\text{FFN}(\mathbf{f})+[{\mathbf{s}}^{\top }00{]}^{\top })$

with $\mathbf{b}=[b_{cx}{b}_{cy}{b}_w{b}_h{]}^{\top }$ and $\mathbf{s}$ the unnormalized 2D reference point (fixed to $(0,0)$ in the original DETR). Because the box is expressed as a displacement from $\mathbf{s}$, the embedding $\mathbf{f}$ already contains the displacements of the four extremities with respect to $\mathbf{s}$ in the unnormalized space. Conditional DETR simply reads that information out and reuses it as a spatial query — no new supervision, no hand-crafted attention prior.

Constructing the conditional spatial query

Two ingredients, $(\mathbf{s},\mathbf{f})\to {\mathbf{p}}_q$:

1. Map the reference point into the same positional space the keys live in, by normalizing and applying the same sinusoidal embedding used for ${\mathbf{p}}_k$:

${\mathbf{p}}_s=\mathrm{sinusoidal}(\mathrm{sigmoid}(\mathbf{s}))$

2. Map the displacement/scale information in the decoder embedding to a linear transformation in that space, $\mathbf{T}=\text{FFN}(\mathbf{f})$ (learnable linear projection + ReLU + learnable linear projection), and apply it:

${\mathbf{p}}_q=\mathbf{T}{\mathbf{p}}_s={\mathbf{\lambda }}_q\odot {\mathbf{p}}_s$

$\mathbf{T}$ is restricted to a diagonal matrix, so it is stored as a 256-dimensional vector ${\mathbf{\lambda }}_q$ and applied by element-wise multiplication — cheap, and empirically on par with a full matrix (see ablations). Because ${\mathbf{p}}_q$ is produced in the same space as ${\mathbf{p}}_k$, the spatial dot product ${\mathbf{p}}_q^{\top }{\mathbf{p}}_k$ is meaningful from the start of training.

Multi-head attention is unchanged ($M=8$ heads): spatial and content queries/keys are projected to each head by separate linear projections, while the value projections stay exactly as in DETR (content only). Because each head projects ${\mathbf{p}}_q$ differently, the heads spread out over different distinct regions.

In the first decoder layer there is no previous decoder embedding, so the implementation falls back to DETR’s behavior with a concatenation instead of an addition: the positional embedding predicted from the object query is concatenated into the query (and key).

What the transformation learns

Visualizing the softmax-normalized ${\mathbf{p}}_q^{\top }{\mathbf{p}}_k$ (spatial), ${\mathbf{c}}_q^{\top }{\mathbf{c}}_k$ (content) and their sum reveals what $\mathbf{T}$ does. Five of the eight heads carry distinct roles — four extremities plus one region inside the box — and the remaining three duplicate them. The spatial maps localize a band overlapping the corresponding box edge; the content maps highlight scattered similar regions (including other instances of the same class); their combination keeps only the extremity highlights. So the transformation has two effects:

• translate the highlight to each of the four extremities and to an interior position, and
• scale the spatial spread — wide bands for large objects, narrow for small ones.

Which means $\mathbf{T}$ encodes not only displacement but also object scale. The interior-region head depends mostly on the spatial map, suggesting a region inside the box already carries enough information for classification.

Reference points

Two options are studied for $\mathbf{s}$: learn it as a per-prediction model parameter, or predict it from the object query, $\mathbf{s}=\text{FFN}({\mathbf{o}}_q)$ (as in Deformable DETR). Both beat DETR’s shared $\mathbf{s}=(0,0)$: 40.7 and 40.9 AP respectively, vs 36.8.

Relation to other fast-converging DETRs

• Deformable DETR replaces global dense attention with sparse deformable attention over $K$ sampled points, and additionally uses a multi-scale, high-resolution encoder. Conditional DETR keeps global dense attention and single-scale features, changing only the query formulation — it is a strictly smaller change and does not address the encoder’s quadratic cost.
• SMCA (spatially modulated co-attention) is the closest concurrent work: it modulates the global cross-attention with hand-crafted Gaussian maps around learned centers. Conditional DETR instead learns the spatial queries from the decoder embeddings, so the attenuation form is not hand-crafted and the resulting maps pick out the four extremities and distinct interior regions.
• The construction is also related to conditional/dynamic convolution (CondConv, dynamic filter networks): there the convolutional kernel weights are predicted from the input; here a linear projection is predicted from the decoder embedding to represent displacement and scaling.

Hyperparameters

Architecture is otherwise identical to DETR; the following is what the paper reports.

Setting	Value
Dataset	COCO 2017 (118K train / 5K val)
Backbones	ResNet-50, ResNet-101, DC5-R50, DC5-R101 (ImageNet-pretrained, batchnorm frozen)
Transformer init	Xavier
Object queries	$N=300$ (DETR uses 100)
Attention heads	$M=8$; decoder repeated 6 times
Prediction FFNs	One after each of the 6 decoder layers, parameters shared across the 6
Optimizer	AdamW, weight decay $10^{-4}$
Learning rate	$10^{-5}$ backbone, $10^{-4}$ transformer
Dropout	0.1
Schedule	50 / 75 / 108 epochs, LR $\times 0.1$ after 40 / 60 / 80 epochs
Classification loss	Focal loss (as in Deformable DETR, not DETR’s cross-entropy)
Box loss	L1 + GIoU, same weights as DETR
Matching	Hungarian bipartite matching, same cost as Deformable DETR
Augmentation	Resize short side $\in [480,800]$, long side $\le 1333$; random crop to a rectangular patch with probability 0.5

Results

COCO 2017 val (Table 1). Conditional DETR at 50 epochs already matches or beats DETR at 500 for the DC5 backbones:

Model	Epochs	GFLOPs	Params	AP	AP${}_S$	AP${}_M$	AP${}_L$
DETR-R50	500	86	41M	42.0	20.5	45.8	61.1
DETR-R50	50	86	41M	34.9	14.4	37.2	54.5
Conditional DETR-R50	50	90	44M	40.9	20.8	44.6	59.2
Conditional DETR-R50	108	90	44M	43.0	22.7	46.7	61.5
DETR-DC5-R50	500	187	41M	43.3	22.5	47.3	61.1
Conditional DETR-DC5-R50	50	195	44M	43.8	24.0	47.6	60.7
Conditional DETR-DC5-R50	108	195	44M	45.1	25.3	49.0	62.2
DETR-DC5-R101	500	253	60M	44.9	23.7	49.5	62.3
Conditional DETR-DC5-R101	50	262	63M	45.0	26.1	48.9	62.8
Conditional DETR-DC5-R101	108	262	63M	45.9	27.2	50.3	63.3

Against other single-scale DETR variants at comparable budgets, conditional DETR-R50 (40.9) beats Deformable DETR-R50-SS (39.4) and UP-DETR-R50 at 150 epochs (40.5); conditional DETR-DC5-R50 (43.8) beats deformable DETR-DC5-R50-SS (41.5). Against the multi-scale, $8\times$-resolution variants (deformable DETR, TSP-FCOS, TSP-RCNN) the single-scale $16\times$ DC5 models come out roughly on par — 43.8 vs deformable DETR-R50’s 43.8, 45.1 at 108 epochs vs TSP-RCNN-R50’s 45.0 at 96 epochs.

Ablations

Reference points (ResNet-50): $\mathbf{s}=(0,0)$ as in DETR → 36.8 AP; learned per-prediction parameter → 40.7; predicted from the object query → 40.9.

How the spatial query is formed (Table 3, ResNet-50). CSQ-P uses only ${\mathbf{p}}_s$ (37.8), CSQ-T only the transformation ${\mathbf{\lambda }}_q$ (37.6), CSQ-C the decoder content embedding $\mathbf{f}$ (37.1), CSQ-I the element-wise product of a transformation predicted from the self-attention output ${\mathbf{c}}_q$ with ${\mathbf{p}}_s$ (40.2). The proposed ${\mathbf{p}}_q={\mathbf{\lambda }}_q\odot {\mathbf{p}}_s$ gives 40.9 — both ingredients are needed, and they must come from the decoder embedding and the reference point.

Isolating the contribution (Table 4, ResNet-50). Starting from DETR’s 34.9: offset regression w.r.t. a learned reference point alone → 35.0; focal loss alone → 35.3; both → 36.9; adding conditional cross-attention → 40.9. So the borrowed Deformable DETR ingredients account for ~2 AP and the conditional cross-attention mechanism for a further +4.0 AP.

Form of $\mathbf{T}$ (Figure 5). Identity (i.e. no learned transformation) 37.8; single learned scalar 39.0; block-diagonal ($32\times 32$ per head) 40.8; full matrix 40.7; diagonal 40.9. A single scalar already helps — narrowing the spatial range to the object’s scale is most of the benefit — and the diagonal form gets the rest at negligible cost.

Limitations

• The encoder is untouched, so the $O(H^2{W}^{2}C)$ cost of global self-attention over the feature map remains. High resolution is still bought via the expensive DC5 backbone rather than a multi-scale encoder, and the paper explicitly leaves combining conditional cross-attention with a multi-scale encoder to future work.
• Convergence is $6.7$–$10\times$ faster, not the $\ge 10\times$ plus small-object gain that multi-scale approaches deliver; single-scale features still cap AP${}_S$ (20.8 for R50 at 50 epochs).
• Three of the eight attention heads are duplicates of the other five, so the head budget is not fully used.
• The conditional spatial query only sharpens the decoder cross-attention; the slow, unstable Hungarian matching assignment early in training is not addressed (that is the target of the later DN-DETR / DINO line).

See also

• DETR — the base architecture whose cross-attention this reformulates
• Deformable DETR — the other main answer to DETR’s slow convergence, via sparse multi-scale attention
• YOLOv10 — NMS-free end-to-end detection from the YOLO line
• Implementing Research Papers