RefT

RefT (Reference Twice) is a unified baseline for Few-Shot Instance Segmentation (FSIS) that abandons the Region Proposal Network entirely and builds on a mask-transformer detector (Mask2Former) instead.

• Paper: Yue Han, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Yong Liu, Lu Qi, Xiangtai Li, Ming-Hsuan Yang (Zhejiang University APRIL Lab / Youtu Lab Tencent / TikTok Singapore / UC Merced), Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation, arXiv:2301.01156. The PDF in this vault is v3, 8 August 2024, typeset in the IEEE journal style; it names no venue on the paper itself, so it is cited here as an arXiv preprint.
• Core idea: the support branch should be consulted twice.
• First reference (feature level): masked-pooled class prototypes reweight the query feature map.
• Second reference (query level): the support branch’s own object queries are cross-attended into the query image’s object queries.
• Real contribution: the second reference. The paper shows empirically that after base training alone, support object queries already encode both the localization and the categorization of novel-class instances, so a plain multi-head cross-attention is enough to transfer that knowledge, with no elaborate spatial-correlation module required.

Problem setup: FSIS, gFSIS, iFSIS

The object classes are split into a disjoint base set $C_{\text{Base}}$ and novel set $C_{\text{Novel}}$, with $C_{\text{Base}}\cap C_{\text{Novel}}=\varnothing$ and $C_{\text{Base}}\cup C_{\text{Novel}}=C_{\text{All}}$. A model is trained on abundant samples of $C_{\text{Base}}$, then fine-tuned with $K$ samples per class of $C_{\text{Finetune}}$, and finally must detect and segment every instance of $C_{\text{Test}}$ in a query image. The three settings differ only in which classes appear in fine-tuning and in testing:

Setting	$C_{\text{Finetune}}$	$C_{\text{Test}}$	The difficulty
FSIS (standard)	$C_{\text{Base}}\cup C_{\text{Novel}}$	$C_{\text{Novel}}$	pure novel-class generalization
gFSIS (generalized)	$C_{\text{Base}}\cup C_{\text{Novel}}$	$C_{\text{Base}}\cup C_{\text{Novel}}$	novel classes without losing base classes
iFSIS (incremental)	$C_{\text{Novel}}$ only	$C_{\text{Base}}\cup C_{\text{Novel}}$	as above, but base data is inaccessible — catastrophic forgetting

iFSIS is the hardest and the practically motivated one: base training data may be unavailable at adaptation time for data-security or storage reasons. Metrics are COCO-style AP at IoU $0.5:0.95$ reported separately for novel and base classes, nAP and bAP.

This is a strictly harder problem than the few-shot semantic segmentation of HSNet. There, each episode is a class-agnostic binary foreground/background problem: one class, one mask, no instances, and the model never has to say which class it found. FSIS requires (i) separating instances, (ii) emitting an actual class label from a fixed vocabulary, and (iii) in the gFSIS/iFSIS variants, keeping the 60 base classes working at the same time. HSNet’s prototype-free dense-correspondence trick has nothing to say about the classification half of that problem, which is exactly where RefT’s ablations show the errors concentrate.

Motivation: two failure modes of RPN-based FSIS

1. RPN overfitting suppresses novel objects. Existing FSIS methods (Meta R-CNN, FGN, MTFA, iFS-RCNN) are Mask R-CNN derivatives. The RPN is trained on base classes with enough data to learn a base-class objectness rather than generic objectness, and consequently scores novel-class regions as background. Supervision collapses before the detection head is ever reached. Mitigations in the literature — mining additional samples, unfreezing more parameters — add data or parameters rather than removing the cause.

2. Dual-branch prototypes need complex spatial correlation. The second family aggregates support features into class prototypes and injects them into the query branch (channel-wise multiplication in Meta R-CNN, three-stage aggregation in FGN). Pooling into a prototype destroys spatial structure, so later work replaced it with dense query–support spatial-correlation attention — expensive, and requiring careful design to avoid under-exploiting the support cues.

RefT’s answer to (1) is to swap the meta-architecture: a query-based mask transformer has no proposal stage to overfit. Its answer to (2) is to observe that the object queries produced by the support branch of that same transformer are a far better carrier of class information than any pooled prototype, because they already entangle “where the instance is” with “what class it is”.

The two empirical findings

Both come from probing a Mask2Former after base training only, before any novel fine-tuning:

• Support query localization (Tab. 2). Feeding novel-class support images through the model and comparing predicted masks to ground truth: with the top-$k$ object queries per image, $k=10$ gives bIoU $0.84$ / nIoU $0.79$ on base/novel classes — identical before and after novel fine-tuning. At $k=30$ it is $0.70/0.64$ (base training) versus $0.75/0.71$ (fine-tuned), at $k=50$, $0.54/0.47$ versus $0.59/0.55$. So the model can already localize and segment novel objects; only the class label is missing.
• Support query categorization (Fig. 3). Cosine-similarity maps among support object queries of the 20 COCO novel classes are already block-diagonal before fine-tuning, and become sharply more so after. Object queries of the same class cluster; a t-SNE of the same queries (Fig. 8) shows the same effect.

The complementary negative result is the baseline’s error profile: a fine-tuned Mask2Former systematically misclassifies semantically similar classes (train predicted as refrigerator, Fig. 2) while its masks are fine. FSIS on a mask transformer is a classification problem, not a localization problem — which is why guidance should be injected at the query level where categorization lives.

Architecture

The backbone is ResNet-50 (Swin-T/S/B in the scale-up experiments) and the meta-architecture is Mask2Former: a pixel decoder built from multi-scale deformable-attention encoder layers (the Deformable DETR MSDeformAttn operator; the figure shows five stacked stages feeding an FPN) followed by a Transformer decoder that turns $Q$ learnable object queries into per-instance class logits and mask embeddings, with masked attention restricting each query’s cross-attention to its own predicted mask region. The decoder is the same set-prediction machinery as DETR and its descendants (DINO, DAB-DETR): Hungarian matching, no NMS, no anchors — hence no RPN to overfit.

Two branches share every weight:

• Query branch: the query image $I_Q$ at $1024\times 1024$ → backbone → CAL → pixel decoder → Transformer decoder → object queries $q_Q\in {\mathbb{R}}^{Q\times D}$.
• Support branch: a support set $S$ of $N\times K$ examples (an $N$-way $K$-shot episode) at the cheaper $320\times 320$ → the same path, but every support image is masked with its ground-truth instance mask so background and other instances are dropped, giving multi-scale support features $F_S$ and object queries $q_S\in {\mathbb{R}}^{NKQ\times D}$ that correspond exactly to support-class instances.

CAL — the Class Adaptation Layer is the paper’s name for the input projection layer sitting between the backbone and the pixel decoder (a single convolution mapping the channel dimension). The paper argues, and Tab. 17 confirms, that this layer is class-specific: it is the crucial thing to fine-tune when adapting a DETR-like model to new classes, and the crucial thing to protect when adapting incrementally.

Crucially, support class prototypes and support object queries are computed once for all classes and reused for every query image — a single forward pass covers all $N$ classes, unlike prior dual-branch methods that need one forward pass per class. Comparing all classes simultaneously is also what mitigates the confusion between semantically similar classes.

Reference Twice

First reference — feature-level enhancement

Multi-scale features from the query image $F_Q=\{x_Q^l{\}}_{l=1}^L$ and from the supports $F_S^{nk}=\{x_S^l{\}}_{l=1}^L$ ($n=1,\dots ,N$; $k=1,\dots ,K$; $L$ feature levels) are first mapped into a common space by the weight-shared multi-scale deformable attention of the pixel decoder, giving $F_Q^{\prime }$ and $F_S^{\prime }$. Support instances are separated from background and from other instances by mask pooling with the ground-truth masks at each scale, then averaged over shots and scales into one adaptive class prototype per class:

$c^n=\frac{1}{KL}{\sum }_{k=1}^K{\sum }_{l=1}^L\mathrm{MaskPool}\left(F_S^{nk}\right),n=1,\dots ,N.$

With the prototype matrix $P=[c^1,\dots ,c^N]\in {\mathbb{R}}^{N\times d}$, a multi-head attention generates a channel reweighting matrix

$R=\mathrm{softmax}\left(F_Q^{\prime }{P}^{\top }\right)\sigma (P),$

(linear projections omitted for clarity; $\sigma$ is the sigmoid) and the query features are rescaled along the channel dimension:

$F_Q^{\text{Enhanced}}=F_Q^{\prime }\odot R.$

The query feature map is thus coarsely filtered by support categories: channels that matter for the classes present in the episode are amplified. Because prototypes for all support classes enter one softmax, the operation is “one-forward-all-class” rather than the “one-forward-one-class” of previous dual-branch methods. Adaptivity comes from the episode composition: the classes present in the support set adapt to the classes appearing in the query image, and negative classes (classes absent from the query image) are explicitly sampled into the support set so training mimics inference, where prototypes for all classes — positive and negative — are available. The prototypes themselves are supervised with a cosine-similarity cross-entropy loss encouraging each to fall into its own category.

Second reference — query-level enhancement

Feature-level aggregation cannot exploit instance-wise information, because a pooled prototype is a class-discriminative vector, whereas object queries carry entangled classification and positional information. The second reference operates directly on queries.

Support object queries first pass a filter: their predicted masks are compared to the ground-truth support masks by IoU, and only the top $k$ of the $Q$ queries per support image are kept. This is both a computation saver and a quality filter — it is exactly the top-$k$ regime validated by the support query localization probe above. Then a multi-head cross-attention matches query object queries against the retained support object queries:

$q_Q^{\text{Enhanced}}=\mathrm{softmax}\left(q_Q\mathrm{TopK}(q_S{)}^{\top }\right)\mathrm{TopK}(q_S).$

A multi-head self-attention and one feed-forward layer follow, adapting the result to the class and mask prediction heads. That is the whole mechanism: no adaptive convolution, no auxiliary contrastive objective, no spatial correlation volume. Tab. 6 tests both alternatives — replacing the cross-attention with adaptive convolution ($17.4\to 16.8$ nAP) or adding an auxiliary classification loss over concatenated query/support queries ($17.4\to 17.5$ nAP, $36.0\to 35.2$ bAP) — and neither helps.

Class-Enhanced Base Knowledge Distillation for iFSIS

The problem this solves is specific to DETR-like models. In iFSIS only novel-class data is available. Freezing everything but the classification head — the RPN-era recipe of iMTFA and iFS-RCNN — does not work here, because CAL is class-specific: with CAL frozen the model barely learns novel classes at all ($2.6$ nAP versus $17.4$ with CAL fine-tuned, Tab. 12). But fine-tuning CAL on novel-only data causes catastrophic forgetting of the base classes. Incremental-DETR attacks this with feature distillation plus logits distillation plus selective search for pseudo-labels; RefT argues CAL is the only place the distillation is needed, and drops the rest.

During iFSIS fine-tuning, only CAL, the object queries and the class head are unfrozen. An input image may contain novel-class foreground, base-class foreground and background, but only the novel annotations exist. So the ground-truth novel masks are used to mask novel objects out, and distillation is applied only to the remainder (base foreground + background):

${\bar{F}}^{\text{freeze}}=(1-{\sum }_{p=1}^P{M}_p^{\text{gt}})F^{\text{freeze}},{\bar{F}}^{\text{finetune}}=(1-{\sum }_{p=1}^P{M}_p^{\text{gt}})F^{\text{finetune}},$

where $M_p^{\text{gt}}$ is the ground-truth mask of the $p$-th novel instance and $F^{\text{freeze}},F^{\text{finetune}}$ are the CAL output features of the model frozen after base training and of the model being fine-tuned.

Base-class object locations are unknown, but base-trained features already indicate them approximately, so the relative prominence of the frozen feature map serves as a spatial and channel reweighting map — the absolute mean value of each channel or pixel, normalized by a softmax:

$R_k^C=C\cdot \mathrm{softmax}\left(\frac{1}{HW}{\sum }_{i=1}^H{\sum }_{j=1}^W\left|F_{k,i,j}^{\text{freeze}}\right|\right),R_{i,j}^S=HW\cdot \mathrm{softmax}\left(\frac{1}{C}{\sum }_{k=1}^C\left|F_{k,i,j}^{\text{freeze}}\right|\right).$

The re-weighted base-class feature distillation loss is then

${\mathcal{L}}_{FD}={\sum }_{k=1}^C{\sum }_{i=1}^H{\sum }_{j=1}^W{R}_k^C{R}_{i,j}^S{\left({\bar{F}}_{k,i,j}^{\text{freeze}}-{\bar{F}}_{k,i,j}^{\text{finetune}}\right)}^2,$

and a coefficient distillation loss keeps the reweighting maps themselves from drifting:

${\mathcal{L}}_{CD}={\sum }_{i=1}^H{\sum }_{j=1}^W{\left(R_{i,j}^{\text{finetune}}-R_{i,j}^{\text{freeze}}\right)}^2+{\sum }_{k=1}^C{\left(R_k^{\text{finetune}}-R_k^{\text{freeze}}\right)}^2.$

The final Class-Enhanced Base Class Distillation (CE-BCD) loss is

${\mathcal{L}}_{CE\text{-}BCD}={\mathcal{L}}_{FD}+\lambda {\mathcal{L}}_{CD},$

with $\lambda$ balancing the two terms. The paper does not report a numerical value for $\lambda$.

Training pipeline

Two stages, both episodic (following Meta R-CNN’s protocol):

1. Base training. Train the whole model on $C_{\text{Base}}$ with abundant data, standard Mask2Former classification and mask losses, plus the cosine-similarity cross-entropy on the adaptive class prototypes. 50 epochs.
2. Novel fine-tuning. Freeze most parameters and fine-tune only the class-specific ones: CAL, the object queries, and the class head. Nothing else is modified. For iFSIS, the class head is additionally replaced by a cosine-similarity classifier (as in iMTFA) and the CE-BCD loss is applied to CAL.

An episode $E_i=(I_Q^i,S^i)$ samples a query image whose objects belong to $N_{\text{pos}}$ classes in $C_{\text{train}}$, plus $N_{\text{neg}}$ randomly added classes absent from the query image, giving an $N$-way $K$-shot episode with $N=N_{\text{pos}}+N_{\text{neg}}$ varying between $N_{\text{pos}}$ and $N_{\text{train}}$. To reduce cost only positive-class support features and queries are actually computed; negatives are reused from previous iterations.

$K$-shot aggregation. Shots are merged at two different points, which is worth noting: at the feature level they are averaged into a single prototype per class (the $\frac{1}{KL}{\sum }_k{\sum }_l$ above), while at the query level they are not averaged — all $NK$ sets of top-$k$ support object queries are concatenated and cross-attended jointly. Contrast HSNet, which aggregates $K$ shots only at the prediction level by voting over $K$ forward passes.

At inference, class prototypes and support object queries are computed once and for all and cached; each query image is a single forward pass covering every class.

Hyperparameters

Reported configuration (Sec. 4, “Implementation Details and Strong Baselines” and “Training and Inference Details”):

Setting	Value
Main module	Mask2Former
Backbone	ResNet-50 (ImageNet-pretrained); ResNet-101 and Swin-T/S/B in scale-up experiments
Dataset	MS-COCO 2014, 60 base / 20 novel classes (the novel 20 intersect PASCAL VOC); LVIS for generalization; FSS-1000 for cross-domain
Shots	$K\in \{1,3,5,10,30\}$ on COCO; $K\le 10$ on LVIS; $K\in \{1,3,5\}$ on FSS-1000
Base training epochs	50
Batch size	8
Hardware	4 × RTX 3090
Optimizer	AdamW
LR schedule	step schedule; initial learning rate $1\times 10^{-4}$, “weight decay at the last epoch by 0.05”
Novel fine-tuning	same settings as base training
Unfrozen in fine-tuning	CAL, object queries, class head (all else frozen)
Query image resolution	$1024\times 1024$
Support image resolution	$320\times 320$
Support features for 1st reference	flattened multi-scale features from the first Transformer encoder layer (Enc1)
Pooling in 1st reference	mask pooling with ground-truth masks
Top-$k$ queries in 2nd reference	$k=10$, selected by mask IoU against ground truth
iFSIS class head	cosine-similarity classifier
Episode composition	$N$-way $K$-shot, $N=N_{\text{pos}}+N_{\text{neg}}$, negatives sampled from classes absent in the query image
Evaluation protocol	10 runs with $K$ examples from 10 seeds per class, averaged
Model size / cost	46.4M parameters, 239.4 GFLOPs at $1024\times 1024$ (Mask2Former baseline: 43.7M, 226.2 GFLOPs)

The paper does not report: the number of object queries $Q$ (Mask2Former’s default is 100), the number of fine-tuning iterations or epochs, data augmentation beyond “the same settings” as Mask2Former, the CE-BCD balancing weight $\lambda$, or individual loss weights.

Results

All COCO numbers with ResNet-50 unless noted; nAP = novel-class AP, bAP = base-class AP, at IoU $0.5:0.95$.

FSIS (standard)

Instance segmentation nAP on COCO:

Method	1	3	5	10	30
MRCN-ft-full	0.6	–	1.2	1.9	–
Meta R-CNN	–	–	2.8	4.4	–
MTFA	–	–	6.6	8.4	–
iFS-RCNN	5.5	7.8	9.4	10.2	13.1
Meta-DETR	–	–	8.1	10.1	–
Mask2Former-ft-full	2.5	8.4	12.7	16.7	20.8
Mask2Former w/ CAM	3.9	11.1	13.4	17.5	21.6
RefT	5.1	12.2	14.2	18.4	22.5

The abstract’s headline +8.2 / +9.4 is RefT versus the best RPN-based prior (iFS-RCNN) at 10-shot ($18.4$ vs $10.2$) and 30-shot ($22.5$ vs $13.1$) instance segmentation. Note that most of that gap is the stronger meta-architecture — Mask2Former-ft-full alone already reaches $16.7/20.8$; RefT’s own contribution over that baseline is $+1.7/+1.7$. The detection side (FSOD nAP) reads $5.2/11.3/15.0/19.3/24.0$, substantially better than Meta-DETR at 10/30 shots ($19.0/22.2$) and comparable at 1/3/5.

gFSIS (generalized)

Instance segmentation, R-50:

Method	nAP 1	3	5	10	30	bAP 1	3	5	10	30
MTFA	2.3	–	6.4	8.4	–	29.9	–	31.3	31.8	–
Mask2Former-ft-full	2.9	7.6	8.5	14.7	17.9	36.1	35.6	33.4	36.0	35.2
Mask2Former w/ CAM	5.2	9.3	11.2	16.1	20.3	33.4	33.2	32.9	33.5	33.5
RefT	5.2	10.2	12.4	17.4	21.7	36.3	36.5	34.4	36.0	35.5

RefT gains on novel classes without trading away base classes — bAP is level with or above the plain fine-tuned baseline, whereas the CAM variant loses ~3 bAP. Detection (gFSOD) with R-50 gives nAP $5.2/10.2/13.1/18.6/23.4$ and bAP $38.4/36.5/36.0/37.7/37.2$; with R-101, nAP $5.2/10.8/14.1/18.9/23.6$, bAP $38.5/36.4/36.2/37.7/37.4$.

iFSIS (incremental)

Instance segmentation, cosine classifier:

Method	nAP 1	3	5	10	30	bAP 1	3	5	10	30
iMTFA	2.8	–	5.2	5.9	–	25.9	–	22.6	21.9	–
iFS-RCNN	4.0	–	8.8	10.1	–	–	–	36.3	36.3	–
Mask2Former w/ Cos	3.1	6.3	9.0	12.4	18.5	29.0	28.4	28.4	28.7	28.6
RefT w/ Cos	3.1	6.0	8.8	11.1	17.7	37.0	36.3	35.3	35.2	32.1

This is the clearest illustration of what CE-BCD buys: RefT trades ~1 nAP for +6 to +8 bAP over the Mask2Former baseline, i.e. it stops forgetting. On the detection side (iFSOD) RefT reaches bAP $32.2/32.0/31.8/33.4/33.2$ against Incremental-DETR’s $27.3$ at 10-shot.

LVIS and cross-domain

LVIS ($K\le 10$; frequent + common classes as base, rare as novel; fAP/cAP/rAP): RefT reaches rAP $23.5$ on FSIS (+1.5 over Mask2Former-ft-full) and $22.1$ on gFSIS (+0.9). On iFSIS it retains base knowledge dramatically better: cAP $26.9$ vs $19.9$ (+7.0) and fAP $28.1$ vs $22.0$ (+5.9), at comparable novel performance.

COCO → FSS-1000 cross-domain (train on the 60 COCO base classes, fine-tune and test on 100 FSS-1000 classes with instance labels, disjoint from COCO base): RefT $82.7/83.3/84.1$ at $K=1/3/5$, versus Mask2Former-ft-full $81.9/82.5/83.4$, iFS-RCNN $81.5/82.2/83.6$ and MRCN-ft-full $79.7/80.3/81.1$.

Scale-up (gFSIS, instance segmentation nAP at $K=1/5/10$): Swin-T $5.2/15.4/18.5$, Swin-S $5.0/19.4/22.7$, Swin-B $7.1/20.7/24.8$; bAP up to $40.2/38.7/39.2$ — the approach keeps improving with backbone capacity rather than overfitting.

Ablations

All on COCO minival, $K=10$, R-50, gFSIS instance segmentation unless noted.

Reference twice, component by component (Tab. 9):

Baseline	+1st reference	+2nd reference	nAP	bAP
Mask2Former-ft			14.7	36.0
Mask2Former-ft	✓		15.6	36.3
Mask2Former-ft		✓	16.3	35.5
Mask2Former-ft	✓	✓	17.4	36.0

The two are complementary but asymmetric: the first (feature-level) reference contributes $+0.9$ nAP and $+0.3$ bAP — it is backbone feature aggregation, and since both branches share the backbone, base-class features in the support branch remain prominent, so nothing is lost. The second (query-level) reference contributes $+1.6$ nAP but $-0.5$ bAP: it pushes the model to rely on support object queries for classification, reducing its bias toward base classes.

Feature-level vs. query-level referencing of the same information (Tab. 11). Implementing the second reference in an image-based way — using the first reference’s adaptive class prototypes as guidance with cross-attention — gives $16.3$ nAP / $35.7$ bAP, versus $17.4/36.0$ for the query-level version. $+1.1$ nAP purely from operating on object queries rather than prototypes; this is the direct evidence for the paper’s central claim.

First reference vs. Meta-DETR’s CAM (Tab. 16). Swapping in Meta-DETR’s Correlational Aggregation Module costs $3.7$ bAP with RefT’s support set and loss ($36.0\to 32.3$), and $9.2$ bAP when additionally aligned to Meta-DETR’s one-class-per-pass support organization and sigmoid binary matching loss ($36.0\to 26.8$). CAM handles class matching itself and leaves the class head class-agnostic, so base-class feature representations are never stored in the head.

Other design choices:

• Support features for the first reference (Tab. 10): Res3 $17.2$, Res4 $17.0$, Res5 $16.4$, Enc1 (flattened multi-scale features from the first encoder layer) $17.4$. Larger-scale features detect small objects better; multi-scale adds another $0.2$.
• Pooling (Tab. 13): GAP $17.0/35.4$, RoIAlign $17.2/35.3$, MaskPool $17.4/36.0$ — accuracy of the extracted region tracks performance directly.
• Query selection in the second reference (Tab. 14): sorting by mask IoU ($17.4/36.0$) beats sorting by score ($17.2/35.8$) slightly, since at small $k$ object queries exhibit high accuracy either way.
• Number of selected queries (Tab. 15): $k=3\to 17.4/35.8$, $k=10\to 17.4/36.0$, $k=50\to 17.5/35.8$, $k=100\to 16.8/35.5$. Stable for $k\le 50$, degrading beyond — matching the mask-quality probe of Tab. 2.
• Which layers to fine-tune (Tab. 17): pixel decoder + queries/head $1.8$ nAP; + Transformer decoder $2.5$; + Res5 $11.7$; CAL + queries/head $17.4/36.0$. Fine-tuning more parameters on $K$ samples overfits badly (bAP $-18.8$); CAL alone generalizes best on both novel ($+5.7$) and base ($+3.1$).
• CE-BCD ablation, iFSIS (Tab. 18): plain RefT $12.4$ nAP / $28.7$ bAP; L2 weight regularization on CAL $11.5/29.4$; Incremental-DETR’s BKD loss $10.1/33.2$ (+3.8 bAP, $-1.4$ nAP); Class-Enhanced BKD $11.1/35.2$ (+6.5 bAP, $-1.3$ nAP).
• iFSIS class head (Tab. 19): a Bayesian logit classifier (iFS-RCNN’s) gives $12.7/36.0$ over the cosine classifier’s $11.1/35.2$ — better, but the gain is smaller than in region-based methods, which the authors read as headroom for DETR-like models in iFSIS.
• Transplanting RPN-era tricks into Mask2Former (Tab. 20): DCNet’s DRD module, FCT, a cosine head and iFS-RCNN’s Bayesian head all give improvements slightly smaller than or comparable to RefT’s query-level aggregation, and are orthogonal to it.

Without re-training (Tab. 21). Adding a learnable support class embedding to the support-branch query features makes the class head predict “does this object belong to the given support class?” instead of a fixed category, which turns RefT into a re-training-free detector functionally close to FS-DETR. Detection nAP $6.6/9.7/11.3/12.7/13.5$ at $K=1/3/5/10/30$ with bAP held near $34$; class-agnostic training helps a lot in low-shot but gains flatten as shots increase.

Cost (Tab. 23, one RTX 3090, batch size 1, $1024\times 1024$): RefT trains at $0.31$ s/iteration with $5655$ MB VRAM and infers at $0.11$ s with $4705$ MB, versus Mask2Former-ft $0.19$/$0.10$ s and Meta Faster R-CNN $0.97$/$0.92$ s. The extra training cost is one forward pass over the sampled support set; at inference the support set is preprocessed and cached, so RefT is essentially as fast as its baseline, and the small support resolution ($320^2$) keeps memory modest. $+1.7$ nAP costs $+5.8\%$ GFLOPs and $+6.2\%$ parameters.

Limitations

• One-shot is where it fails. The paper’s own stated limitation: a single shot makes the reference-twice guidance unreliable because the support features deviate from the true class distribution. At $K=1$ RefT is beaten by iFS-RCNN on FSIS ($5.1$ vs $5.5$ nAP) and by iFS-RCNN/iMTFA on iFSIS — the gains only materialize from 3 shots up.
• Misclassification remains the dominant error. Even with both references, the qualitative results (Figs. 10–12) show masks are usually right while class labels are often wrong; the second reference reduces but does not eliminate confusion among semantically similar classes.
• Novel/base trade-off is not eliminated, only softened. In iFSIS, CE-BCD recovers base classes at a consistent $\sim 1$ nAP cost, and the query-level reference itself costs $0.5$ bAP.
• Much of the headline gain is the meta-architecture. Against the fair Mask2Former-ft-full baseline the modules are worth $+1.7$ nAP, not $+8.2$; the paper is honest about this and frames itself as a baseline, not a peak.
• Support masks are required at both levels. Mask pooling in the first reference and the IoU-based top-$k$ filter in the second both consume ground-truth support masks, so RefT cannot be prompted with boxes or points the way SAM or SAM3 can.
• The class head is class-specific. Out of the box RefT cannot handle the “without re-training” / open-vocabulary setting; it needs the support-class-embedding modification, whose accuracy then saturates with more shots.
• iFSIS numbers still trail region-based methods on novel classes at low shot, and the Bayesian-head ablation suggests the DETR-like classification head is the remaining bottleneck.

See also

• HSNet — the few-shot semantic segmentation counterpart: class-agnostic binary masks from dense 4D correlation instead of instance masks plus class labels, and $K$-shot aggregation by voting rather than by prototype/query fusion.
• Deformable DETR — the MSDeformAttn operator that RefT’s shared pixel decoder (and Mask2Former’s) is built from, and the source of its multi-scale encoder.
• DETR — the set-prediction formulation, object queries and Hungarian matching that make an RPN-free FSIS baseline possible in the first place.
• DINO and DAB-DETR — the same object-query line, showing how much structure a query can be made to carry; RefT’s finding that support queries already encode class and location is the few-shot analogue.
• PerSAM — the extreme opposite trade-off for “segment this concept from one example”: training-free personalization of a frozen model from a single image and mask, with no base/novel split and no fine-tuning at all.
• SAM and SAM3 — foundation-model answers to the same problem; SAM 3 in particular accepts exemplar boxes as a concept prompt, which is the few-shot setting recast as promptable segmentation over a huge pre-trained vocabulary.
• Your ViT is Secretly an Image Segmentation Model — argues the pixel decoder and Transformer decoder RefT inherits are largely removable given a strong enough ViT.
• Implementing Research Papers — the unreported $\lambda$, query count and fine-tuning schedule here are exactly the gaps that decide whether a reproduction lands.