DE-ViT

DE-ViT (“Detect Everything with Few Examples”) is a few-shot and one-shot object detector that localizes and classifies novel-category objects from only a handful of example images, without any finetuning at test time.

• Paper: Xinyu Zhang, Yuhan Liu, Yuting Wang and Abdeslam Boularias (Rutgers University), Detect Everything with Few Examples, CoRL 2024, arXiv:2309.12969.
• Backbone: a frozen DINOv2 ViT (not to be confused with the unrelated DETR-line DINO detector).
• Core idea: replaces both the prototype-classifier and the dense-matching strategies used by prior meta-learning detectors with two new components:
  • a region propagation network that localizes objects by refining a mask instead of regressing box coordinates;
  • a feature subspace projection that uses class prototypes as a projection basis rather than as classifier weights.
• Headline result: new state-of-the-art results on few-shot Pascal VOC, COCO and LVIS, and on one-shot COCO, while also being faster at inference than the methods it displaces.

Fig. 1: DE-ViT detects YCB objects it has never seen labels for, given only a few example images per class at inference time.

Motivation: finetuning is the bottleneck of few-shot detection

Few-shot object detection (FSOD) splits classes into a set of base classes ${\mathcal{C}}_{base}$, seen with abundant labels during training, and novel classes ${\mathcal{C}}_{novel}$, for which only $k$ labeled support images are available at test time. Most prior FSOD methods rely on finetuning the detector (or at least its classifier head) on the novel-class support set, with complicated multi-stage procedures — LVC, one of the strongest prior baselines, needs more than ten finetuning stages of self-training and pseudo-labeling. This is both slow and impractical for a robot that needs to recognize a new object class instantly, e.g. from a handful of photos.

The natural alternative is to lean on a strong pretrained ViT backbone (frozen DINOv2 features are highly semantic and class-general) and avoid finetuning entirely. The paper shows this is not simply a matter of swapping the backbone in an existing pipeline: naively regressing bounding boxes from frozen DINOv2 features gives poor localization (Appendix C: $\text{nAP75}=14.6$), because pretrained ViT features carry rich semantics but lack fine-grained coordinate information; and unfreezing the backbone to learn this information during base-class training leads to complete overfitting on base classes and an accuracy collapse on novel classes (observed when plugging DINOv2 into Meta R-CNN, a standard prototype-based FSOD baseline). DE-ViT’s two contributions directly target these two failure modes: region propagation supplies localization without gradient updates to the backbone, and feature subspace projection supplies classification robust to base/novel overfitting, again without finetuning.

Fig. 2: (a) prototypes as classifier weights — simple but inaccurate. (b) dense query/support feature matching — accurate but computationally heavy and usually still needs finetuning. (c) DE-ViT: prototypes as a projection basis, followed by a region propagation network; no finetuning at all.

Architecture

DE-ViT is a two-stage detector in the general style of Mask R-CNN / Meta R-CNN, but with both stages redesigned around a frozen backbone:

1. A frozen DINOv2 ViT backbone (evaluated at ViT-S/14, ViT-B/14 and ViT-L/14 sizes) extracts dense features $h_{vit}\in {\mathbb{R}}^{D\times H\times W}$ from the query image. The backbone is never updated, at either base-class training or novel-class inference time.
2. An off-the-shelf class-agnostic region proposal network (RPN), trained separately per dataset using only base-class boxes, produces object proposals. Class-agnostic proposals are used because they are known to generalize well to novel classes, unlike a class-specific RPN.
3. Feature subspace projection (Sec. 2.4) projects the ViT features into a class-prototype-defined subspace instead of feeding raw ViT features (or a learned classifier) into the rest of the network.
4. A region propagation network built from stacked Propagation Layers (PL) (Sec. 2.2) takes each proposal, expands it by a fixed ratio, and iteratively refines an object mask within that expanded region.
5. A learnable spatial integral layer (Sec. 2.3) converts the final propagated mask into bounding-box coordinates, replacing the usual non-differentiable, ground-truth-mask-dependent “min/max foreground pixel” conversion.
6. The propagated mask also doubles as a spatial attention map: it weight-averages the propagated features over height and width, and a linear layer maps the pooled feature to a class score.

Only the RPN, the propagation layers, the spatial integral layer and the final linear classification head are trained (on base classes only); the ViT backbone stays frozen throughout, and the entire pipeline is applied unchanged to novel classes at test time — DE-ViT has 23M trainable parameters in total (compare: OWL-ViT 433M, F-VLM 25M trainable but 445M total).

Fig. 3: Left — a proposal box is expanded and its initial mask region is refined by a stack of Propagation Layers (PL), each taking all previous regions and features as input. Top right — inside a PL block: region and feature updates, the spatial integral layer producing the box, and weighted-average pooling into a class score. Bottom right — feature subspace projection: prototypes dot-product with ViT features, channel-reordered and ROI-aligned into subspace features.

Key methodology

Region Propagation Network and the Propagation Layer

Given a proposal box, DE-ViT extracts an initial mask region — a solid square placed inside the box — and expands the proposal by a fixed ratio to delimit the area the mask is allowed to grow into. This expanded region is then refined by a stack of Propagation Layers (PL), a new module the paper proposes as the central building block of the network. Each PL block behaves like a small detection network in its own right and can be stacked (3 blocks for COCO/Pascal VOC, 5 for LVIS) — the paper’s ablation shows accuracy improving consistently up to 5 blocks and saturating shortly after.

The $t$-th PL block takes all previous output regions $r_{0:t-1}\in {\mathbb{R}}^{t\times K\times K}$ and the previous block’s features $h_{t-1}\in {\mathbb{R}}^{S\times K\times K}$ as input ($K$: spatial size of the feature map, $S$: number of channels), and produces an updated region $r_t$, updated features $h_t$, a bounding box $b_t$, and a class score $c_t$:

$h_t=f_{\text{update},t}(\text{concat}(r_{0:t-1},h_{t-1});\theta ),h_{t,region}=f_{\text{region},t}(h_t;\theta )$
$r_t=\sigma (h_{t,region}),b_t=f_{\text{integral},t}(h_{t,region};\theta )$
$c_t=f_{\text{class},t}(\text{WeightedAvgPool}(h_t,r_t);\theta )$

where $f_{\text{update},t}$ is a conv-BN-ReLU block, $f_{\text{region},t}$ a single-channel conv block predicting mask logits, $\sigma$ the sigmoid, $f_{\text{integral},t}$ the spatial integral layer, and $f_{\text{class},t}$ a linear head. During training, the class score $c_t$ and box $b_t$ are supervised with focal loss and L1 regression loss respectively, and the region logits $r_t$ with a combination of BCE and Dice loss against ground-truth-object masks restricted to the expanded proposal.

Unlike conventional bounding-box regression on ViT features — which the paper shows performs poorly precisely because raw ViT features lack coordinate information — mask-based region propagation localizes objects purely through iterative refinement of a spatial mask, sidestepping the missing-coordinates problem entirely.

Learnable Spatial Integral layer

Converting a predicted mask to a bounding box is normally done as post-processing, by taking the top-left/bottom-right extent of the foreground pixels — a step that is non-differentiable, sensitive to outlier pixels, and (in instance segmentation networks) assumes ground-truth masks are available, which they usually are not for detection-only annotations. DE-ViT instead makes this conversion a learnable, differentiable layer.

Rather than predicting the absolute box $b^{\text{out}}=(c_w^{\text{out}},c_h^{\text{out}},w^{\text{out}},h^{\text{out}})$ directly, it predicts a relative box $b^{\text{rel}}=(c_w^{\text{rel}},c_h^{\text{rel}},w^{\text{rel}},h^{\text{rel}})\in [0,1{]}^4$ normalized against the expanded proposal $b^{\text{exp}}$:

$(w^{\text{out}},h^{\text{out}})=(w^{\text{exp}}{w}^{\text{rel}},h^{\text{exp}}{h}^{\text{rel}})$
$(c_w^{\text{out}},c_h^{\text{out}})=(c_w^{\text{exp}}-0.5w^{\text{exp}},c_h^{\text{exp}}-0.5h^{\text{exp}})+(c_w^{\text{rel}}{w}^{\text{exp}},c_h^{\text{rel}}{h}^{\text{exp}})$

The relative box center is estimated as the expected position of the spatial softmax of the region logits $h_{region}\in {\mathbb{R}}^{K\times K}$:

$(c_w^{\text{rel}},c_h^{\text{rel}})={\sum }_{i,j=1}^{K,K}\left(\frac{i}{K},\frac{j}{K}\right)\ast \text{softmax}(h_{region}{)}_{ij}$

and the width/height are estimated from row and column sums of the sigmoid-activated region mask, aggregated by sorted, learnably-weighted averaging (order statistics) rather than a hard maximum:

$w^{\text{rel}}={\sum }_{i=1}^K{\sum }_{j=1}^K\frac{\sigma (h_{region}{)}_{(i)j}}{K}{\theta }_i^{\mathbf{w}},h^{\text{rel}}={\sum }_{j=1}^K{\sum }_{i=1}^K\frac{\sigma (h_{region}{)}_{i(j)}}{K}{\theta }_j^{\mathbf{h}}$

with learnable per-rank weights ${\theta }^{\mathbf{w}},{\theta }^{\mathbf{h}}\in {\mathbb{R}}^K$. Intuitively (the paper motivates this with a toy binary-triangle-mask example, Fig. 4) the box’s width and height correspond to the maximum row-sum and column-sum of the mask; using a soft, sorted weighted average instead of a hard max keeps the layer differentiable and robust to noisy mask predictions.

Feature Subspace Projection

The central classification idea is to stop using prototypes as classifier weights (the common approach in prior prototype-based FSOD, which the paper shows is inaccurate — Fig. 2a) and instead use them as the basis of a subspace projection. Prototypes $p_c$ for a class $c$ are built by averaging DINOv2 features cropped to the object’s bounding box over the class’s support images (following Meta R-CNN’s prototype extraction procedure). Given unit-normalized prototypes $p_{\mathcal{C}}\in {\mathbb{R}}^{|\mathcal{C}|\times D}$ and query ViT features $h_{vit}\in {\mathbb{R}}^{D\times H\times W}$, the dot product

$p_{\mathcal{C}}\cdot h_{vit}\in {\mathbb{R}}^{|\mathcal{C}|\times H\times W}$

is a subspace projection with $p_{\mathcal{C}}$ as the basis. Two refinements address the raw construction’s weaknesses: (1) prototypes of a fixed set of background (non-object, “stuff”) classes $\mathcal{B}$ — e.g. sky, road, floor, drawn from COCO-Stuff — are appended to preserve information that class prototypes alone would discard; (2) since a subspace built from $\mathcal{C}$ depends on the (otherwise arbitrary) ordering of classes, a separate subspace is built per class $c$, reordering the remaining classes $\mathcal{C}\setminus c$ by projection magnitude to resolve the permutation ambiguity:

$h_{\text{subspace},c}=\text{concat}(p_c\cdot h_{vit},\text{channel-reorder}(p_{\mathcal{C}\setminus c}\cdot h_{vit}),p_{\mathcal{B}}\cdot h_{vit})$

Ablations (Appendix C, Table A2) show this construction is what makes frozen-backbone FSOD viable at all: raw DINOv2 features fed into Meta R-CNN collapse to $\text{nAP50}=4.5$; projecting onto $p_c$ alone already reaches 26.2; adding background prototypes $p_{\mathcal{B}}$ reaches 38.4; adding the reordered $p_{\mathcal{C}\setminus c}$ channels reaches 39.5. Building a per-class subspace is expensive at inference (linear in $|\mathcal{C}|$), so DE-ViT restricts inference to the $T$ classes whose prototypes are closest to the average proposal feature (a lightweight prototype classifier), with $T=10$ as the default and $T=3$ shown to preserve state-of-the-art accuracy at even lower latency.

Hyperparameters

Transcribed from Table A8 (Appendix C):

Hyperparameter	Value
Max image size	$800\times 1333\times 3$
Batch size	16
Optimizer	SGD
Learning rate	0.002
Warm-up steps	5,000
LR schedule	stepwise decay
LR decay steps	60,000 / 80,000
LR decay gamma	0.1
Training steps	90,000
Evaluation frequency	every 5,000 steps (COCO), every 10,000 steps (LVIS)
ViT backbone (layers, channel size)	small (12, 384), base (12, 768), large (24, 1024)
Channels of propagation network	256
Channels of feature subspace projection ($S$)	256
Classes in feature subspace projection ($T$)	10
Number of PL layers	3 (COCO, Pascal VOC), 5 (LVIS)

Additional training details: focal loss + L1 regression for class score / box at every PL block; BCE + Dice loss for the region mask, with region labels generated from the ground-truth object mask restricted to the expanded proposal; the RPN is trained separately per dataset using base-class boxes only and kept frozen at inference; background prototypes $\mathcal{B}$ are extracted from COCO-Stuff images by default (an ablation shows ADE20k works comparably); prototypes are built equally well from bounding-box or instance-mask annotations of the support images (Table A5: nAP50 43.1 vs 43.0 at 5-shot).

Results

Few-shot COCO 2014 (Table 1): DE-ViT (ViT-L/14) reaches 34.0 nAP / 52.9 nAP50 / 37.0 nAP75 at 10-shot and 34.0 nAP / 53.0 nAP50 / 37.2 nAP75 at 30-shot, without any finetuning, surpassing the prior SoTA LVC (which requires 10+ finetuning stages) by +15 nAP at 10-shot and +7.2 nAP at 30-shot.

Few-shot Pascal VOC (Table 2, averaged nAP50 over 3 novel splits): DE-ViT (ViT-L/14) surpasses the prior SoTA NIFF by +2.0 nAP50.

LVIS (Table 3, box AP): DE-ViT (ViT-L/14) reaches ${\text{AP}}_r=33.6$ (rare classes / the few-shot-relevant metric), outperforming the prior SoTA DiGeo’s ${\text{AP}}_r=16.6$ by +17 points — a particularly large margin on what the paper calls “a highly challenging dataset” due to its 337 novel classes.

One-shot COCO 2017 (Table 4): DE-ViT (ViT-L/14) reaches 59.6 average bAP50 / 28.4 average nAP50, beating the prior SoTA BHRL by +6.0 bAP50 / +2.8 nAP50, despite BHRL being restricted to single-class, single-support-image detection while DE-ViT applies the same general multi-class pipeline.

Against language-based open-vocabulary detectors (Appendix Table A4): DE-ViT outperforms CORA${}^{+}$ by +6.9 AP50 on COCO using only COCO training data (CORA${}^{+}$ additionally uses ImageNet-21k and COCO Captions), and outperforms F-VLM (mask ${\text{AP}}_r$) and OWL-ViT (box ${\text{AP}}_r$) on LVIS, while using far fewer trainable parameters (23M vs. F-VLM’s 25M-of-445M and OWL-ViT’s 433M) and far fewer training epochs (14.4 vs. 118 / 1,800).

Efficiency: DE-ViT with $T=1$ achieves 49.6 nAP50 at 0.22 s/image on COCO 10-shot, already beating all compared prior methods on both speed and accuracy; the paper’s Table A1 shows accuracy climbing to 52.9 at $T=10$ (0.83 s/img), still faster than Meta Faster-RCNN (0.61 s) and CrossTransformer (3 s).

Ablations confirm both novel components are necessary: swapping the region propagation network for a conventional coordinate regressor drops $\text{nAP75}$ from 24.1 to 14.6 (Table A3), and removing feature subspace projection drops $\text{nAP50}$ from 39.5 to 4.5 — a complete accuracy collapse (Table A2).

Real-robot experiment: the authors built a pick-and-place system (Kuka LBR iiwa arm, RGB-D camera) that segments objects with SAM to build prototypes from a few example images, detects and instance-segments novel YCB objects with DE-ViT, and generates grasp poses from the resulting point cloud — with no finetuning, novel objects are usable “instantly.” The system achieves a 97% pick success rate over 100 trials across ten YCB object categories.

Fig. 5: A qualitative trace of the region propagation network on a real query image. The transparent-yellow, yellow and green boxes are the raw proposal, the expanded proposal, and the final predicted box; the three intermediate masks are the outputs of the first three PL blocks, refining from a coarse rectangle to a tight fridge-shaped mask.

Limitations

• RPN quality caps novel-object recall. The region proposal network is trained only on base-class boxes and kept entirely frozen at inference, so its ability to propose regions for novel-class objects is not explicitly optimized; the authors suggest either conditioning the RPN on novel prototypes with the same propagation framework, or replacing the two-stage pipeline with a full transformer detector (e.g. DETR-style) to remove the proposal stage altogether.
• Prototype bias from the pretrained backbone. Because classification is driven entirely by prototype-feature dot products rather than dense spatial matching, DE-ViT’s accuracy is bounded by how well frozen DINOv2 features separate the classes in question; this is expected to degrade on long-tailed or out-of-distribution visual domains where DINOv2 itself underperforms (e.g. the real-robot failure cases mix up “chips” and “tomato can”, visually similar cylindrical packaging never seen at training time).
• Per-class inference cost. Building a genuinely separate feature subspace per class scales linearly with $|\mathcal{C}|$; DE-ViT mitigates but does not eliminate this by restricting full inference to the top-$T$ candidate classes from a lightweight prototype classifier.
• Reported results are box-level (and LVIS mask) detection only; no dense per-pixel segmentation ablation of the propagation network’s mask quality is reported outside the box-conversion pipeline.

See also

• SAM3 — a concept-based promptable detection/segmentation model; a complementary route to open-world detection, prompting with text/exemplars against a large trained model rather than DE-ViT’s frozen-backbone + prototype-subspace approach
• SAM — used in DE-ViT’s real-robot pipeline to segment example images into class prototypes
• HSNet — few-shot segmentation via dense correlation matching between support and query features, the segmentation-side counterpart to the “dense matching network” strategy (Fig. 2b) that DE-ViT explicitly moves away from
• PerSAM — one-shot personalization of SAM via a single reference image and feature-similarity prompting, a different frozen-backbone route to few-shot visual recognition
• RefT — another few-shot/reference-guided detection approach, useful to contrast against DE-ViT’s no-finetuning region-propagation design
• DETR — the end-to-end transformer detection framework the authors point to as a way to remove DE-ViT’s separate RPN stage