Few-Shot Segmentation Comparison

Four methods in this wiki claim to segment a concept from a handful of examples — HSNet, RefT, PerSAM and SAM3 — and a fifth, SAM, is the frozen substrate two of them stand on. They are routinely filed under the same label, and that label is misleading: none of the four solves the same problem, none of them evaluates on a benchmark another one reports, and two of them do not have a notion of “class” at all. “Few-shot segmentation” names a supervision budget, not a task, and papers with the same budget can differ more from each other than from methods with a completely different budget. This page separates the axes those papers conflate, puts the numbers side by side with an explicit statement of which pairs are comparable, and argues that the arrival of SAM did not make the problem easier so much as it moved the problem somewhere else. Every number here is transcribed from the individual articles, which carry the methods, the equations and the full hyperparameters.

What is actually being asked

Strip the four papers to their input/output contract and they diverge immediately:

• HSNet is given one image with a binary mask and a query image, and must return a binary foreground map for the query. It never emits a class label; the “class” exists only in the sampling protocol that decided which support and query images go together.
• RefT is given $K$ annotated instances per class for $N$ classes and must return instance masks with class labels from a fixed vocabulary, while not forgetting the 60 base classes it was trained on. Its errors are almost entirely classification errors; the masks are fine.
• PerSAM is given one image with a mask of one particular object — your dog, not dogs — and must find that instance in new images. There is no class vocabulary, no base/novel split, and nothing is trained (or two scalars are).
• SAM3 is given a noun phrase, or exemplar boxes, or both, and must return every instance matching the concept, with identities across video frames. There is no adaptation step at all: the “few shots” are interactive prompts to an already-trained model.

Three of these have different output types, three have different prompt types, two have no class notion, and only one is trained episodically. Calling all four “few-shot segmentation” hides more than it reveals.

Taxonomy

Axis 1 — task output

Output	Method	Consequence
One binary semantic mask per episode	HSNet	No instances, no labels; multi-class needs one pass per class
Instance masks plus class labels	RefT	Must solve detection, segmentation and classification; base-class retention becomes a separate problem (gFSIS/iFSIS)
One personalized instance	PerSAM	Identity, not category: two identical-looking dogs are a failure mode by construction
All instances of a concept, with identities	SAM3	Needs exhaustivity and calibration, which is why it cannot use mIoU or AP as its metric

This axis alone explains most of the apparent incomparability. HSNet’s mIoU is a per-pixel score on a task with exactly one object category present; RefT’s nAP additionally penalizes every misnamed instance; SAM3’s cgF${}_1$ multiplies mask quality by an image-level recognition coefficient and therefore goes to zero for a model that segments perfectly but cannot tell whether the concept is present.

Axis 2 — what the “shot” is

• An annotated support image + mask. HSNet, RefT and PerSAM all require a mask. It is load-bearing in every case: HSNet uses it to zero out background before correlating (${\hat{\mathbf{F}}}_l^{\text{s}}={\mathbf{F}}_l^{\text{s}}\odot {\zeta }_l({\mathbf{M}}^{\text{s}})$, worth $66.2\to 61.5$ mIoU when removed), RefT uses it for mask pooling and for the IoU-based top-$k$ query filter — which is exactly why RefT cannot be prompted with a box or a point — and PerSAM uses it to define the local feature bank.
• A box exemplar. SAM3’s image exemplars are positive/negative boxes, and PerSAM degrades gracefully to one: relaxing the mask to a box costs it $89.3\to 88.1$ mIoU (and PerSAM-F $95.3\to 94.9$) because it just runs SAM on the box to recover the mask. The same relaxation destroys SegGPT ($94.3\to 36.0$). That gap is the strongest single argument in the vault for building on a promptable foundation model: the prompt type becomes almost free to change.
• A text noun phrase. Only SAM3. This is not a “shot” in the statistical sense at all — it carries no pixels — and it is the axis on which the other three have nothing to say.

Axis 3 — is there a base/novel split?

	Class notion	Split	Evaluation
HSNet	classes only in the sampler	disjoint train/test classes, 4-fold	1,000 sampled episodes per fold
RefT	explicit vocabulary	60 base / 20 novel COCO classes	10 runs × 10 seeds, nAP and bAP
PerSAM	none	none	40 subjects × 5–7 images, PerSeg
SAM3	open vocabulary (4M NPs)	none — the split dissolves	3M media-phrase pairs, 84.4% negatives

The base/novel split is the defining apparatus of the pre-foundation-model literature and it is absent from both post-SAM methods. That is not an oversight — it is the point. A split exists to certify that a model generalizes beyond what it was trained on; once training covers 4M noun phrases or 1.1B class-agnostic masks, the certificate is meaningless and the interesting question becomes calibration on negatives instead. SAM3’s benchmark being 84.4% negative datapoints is the direct replacement for the base/novel split.

Axis 4 — training regime

1. Episodic meta-training — HSNet. The model is trained on episodes drawn from base folds and applied unchanged to novel folds. No test-time optimization whatsoever.
2. Base-train, then fine-tune — RefT. 50 epochs on abundant base data, then fine-tune a deliberately tiny subset (CAL, object queries, class head) on the $K$ shots. Its Tab. 17 is the cleanest evidence anywhere in the vault for why this must stay tiny: unfreezing the pixel decoder, the Transformer decoder and Res5 as well costs $-18.8$ bAP.
3. Training-free on a frozen foundation model — PerSAM. Zero parameters, or two. The PEFT ablation is the striking result: prompt tuning (12K params) and Adapter (196K) both fall below the training-free baseline by overfitting the single shot; LoRA (293K) barely matches it; two constrained scalars beat all of them. In a genuine one-shot regime, parameter count is inversely related to accuracy.
4. Pretrain so hard the problem disappears — SAM3. No adaptation step exists. What was “few-shot learning” becomes “interactive prompting”, and the paper measures it that way: three exemplars are worth $+21.6$ cgF${}_1$ over text alone, with no gradient step anywhere.

Axis 5 — learned vs. frozen

Method	Frozen	Learned	Learnable params
HSNet	ImageNet backbone (VGG16 / R50 / R101)	4D-conv pyramid encoder + 2D decoder	2.6M (identical for every backbone)
RefT	backbone and most of Mask2Former during fine-tuning	CAL + object queries + class head	46.4M total; the fine-tuned subset is a small fraction
PerSAM	all of SAM	nothing / the two mask-blend weights	0 (PerSAM) / 2 (PerSAM-F)
SAM3	— (everything is trained, once, centrally)	PE-L+ 450M vision + 300M text, detector, tracker	0 at adaptation time

Read down that column and the field’s trajectory is obvious: 2.6M → a handful → 0, with the compute displaced from adaptation time into a single enormous pretraining run. HSNet already anticipated the logic in miniature — its Fig. 9 ablation shows a fine-tuned backbone overfits the training folds and loses validation mIoU even at $100\times$ lower learning rate, so it froze the backbone and learned correlations instead. PerSAM’s PEFT table is the same finding one abstraction level up, and SAM 3 is that finding taken to its conclusion.

The comparison table

One row per method. These numbers are not mutually comparable — the benchmark column is not a footnote, it is the reason the table has no “winner” column. Read the next section before drawing any ordering from it.

Method	Year	Task output	Prompt / shot	Backbone	Learnable params	Training regime	Benchmark → headline
SAM	2023 (ICCV)	one mask per prompt, class-agnostic	point / box / mask / (text)	ViT-H, MAE-pretrained	all, once (SA-1B)	90k iters on 1.1B masks; no adaptation	zero-shot instance seg. via ViTDet boxes: COCO 46.5 AP, LVIS 44.7
HSNet	2021 (ICCV)	binary semantic mask	1 support image + mask	R101, frozen	2.6M	episodic meta-training, disjoint folds	PASCAL-5${}^i$ 66.2 mIoU (1-shot); COCO-20${}^i$ 41.2; FSS-1000 86.5
PerSAM	2023 (ICLR 2024)	one personalized instance	1 reference image + mask (or box)	frozen SAM ViT-H	0 / 2	training-free; 10 s on one A100 for PerSAM-F	PerSeg 95.3 mIoU (PerSAM-F); DAVIS17 $\mathcal{J}\&\mathcal{F}$ 76.1; FSS-1000 86.3
RefT	2024 (arXiv)	instance masks + class labels	$K$ support images + instance masks	R50 (Swin-T/S/B scale-up)	46.4M total; CAL + queries + head fine-tuned	50-epoch base training, then novel fine-tune	COCO FSIS 18.4 nAP (10-shot); gFSIS 17.4 nAP / 36.0 bAP
SAM3	2025	all instances of a concept + identities	noun phrase and/or exemplar boxes	PE-L+ (450M vision + 300M text)	0 at adaptation time	~95k + 5k + 190k iters, 4-stage	SA-Co/Gold 54.1 cgF${}_1$; ODinW13 AP${}_{10}$ 71.8; RF100-VL 36.5

SAM is in the table as the substrate, not as a competitor: it does no few-shot adaptation, and its row exists because PerSAM is literally SAM with a synthesized prompt, and because SAM’s own limitation — “unclear how to design prompts expressing semantics; SAM produces masks, not labels” — is the exact gap the two post-SAM rows fill.

Deliberately excluded. SAM2, MobileSAM and Your ViT is Secretly an Image Segmentation Model belong elsewhere in the wiki. SAM 2’s semi-supervised VOS setting (annotate frame 1, propagate) looks one-shot and is not: it propagates an identity through a contiguous video with a memory bank, with no generalization across images, no novel class and no adaptation. It is on this page only as the thing PerSAM beats on DAVIS 2017 without any video training at all ($\mathcal{J}\&\mathcal{F}$ 76.1), which is a statement about how far frozen-feature matching goes, not a few-shot result. MobileSAM is encoder distillation and EoMT is architecture reduction; neither adapts from examples.

Why the numbers do not compare

Being specific about which pairs are and are not valid:

Not comparable, different task. HSNet’s PASCAL-5${}^i$ 66.2 mIoU against RefT’s COCO 18.4 nAP is the comparison people are most tempted to make and the most wrong. mIoU on a binary foreground map, averaged over classes of a fold, is computed on a problem where the model is told one class is present and never has to name it. nAP at IoU $0.5:0.95$ over 20 novel classes penalizes every instance split, every duplicate and every misnamed object. RefT’s own article states the errors concentrate in classification — the half of the task HSNet is not asked to do. A number near 66 and a number near 18 are not “HSNet is 3.5× better”.

Not comparable, different data. HSNet’s COCO-20${}^i$ 41.2 mIoU and RefT’s COCO nAP both say “COCO”, and share nothing else: different splits (4 folds of 20 classes vs. a fixed 60/20 base/novel), different metrics, different episode construction, different image resolutions ($400^2$ vs. $1024^2$ query / $320^2$ support).

Not comparable, no shared axis at all. PerSAM’s PerSeg 95.3 mIoU is a 40-subject identity benchmark of the authors’ own construction, built from DreamBooth/Textual-Inversion source images. It measures “did you find the same teapot again”, not “did you generalize to a novel category”. A 95 on PerSeg and a 66 on PASCAL-5${}^i$ are measurements of different phenomena and their ratio means nothing. Likewise SAM3’s cgF${}_1$: it is $100\cdot {\text{pmF}}_1\cdot \text{IL\_MCC}$, a mask-quality F1 gated by a Matthews correlation coefficient over image-level presence decisions on an 84.4%-negative benchmark. It is not on the same scale as anything else in the table, deliberately — the SAM 3 paper argues AP is unusable over a 4M-phrase label space because most labels appear a handful of times and AP ignores calibration entirely.

Genuinely comparable. Two pairs survive:

1. FSS-1000, 1-shot mIoU: HSNet 86.5 (R101) vs. PerSAM-F 86.3. Same benchmark, same shot count, same metric. This is the single most informative row on the page. A method with 2.6M learnable parameters, trained episodically for ~100 epochs, and a method with two learnable parameters and ten seconds of fitting, land within 0.2 mIoU of each other. Everything HSNet’s 4D-correlation machinery buys over a frozen foundation model plus cosine matching is, on this benchmark, inside the noise. That is the whole argument of the post-SAM era in one comparison, and it is available in the vault without leaving the two articles.
2. Within RefT’s own tables, against Mask2Former-ft-full and iFS-RCNN — because RefT re-ran them. The abstract’s $+8.2/+9.4$ is against the RPN-based iFS-RCNN; against the fair same-meta-architecture baseline the modules are worth $+1.7$ nAP. RefT’s article is explicit about this, and it is the reason the headline number should not be moved into a cross-method table without the baseline beside it.

Two near-misses worth naming. RefT’s COCO→FSS-1000 numbers ($82.7/83.3/84.1$ at $K=1/3/5$) are not comparable to HSNet’s and PerSAM’s FSS-1000 numbers despite the shared dataset name: RefT evaluates instance labels on 100 classes after base-training on COCO, a different protocol. And PerSAM-F’s LVIS-92${}^i$ 18.4 mIoU sits suspiciously close to RefT’s LVIS rAP 23.5 — different metric, different split, no relation.

The mechanism axis

Strip the papers to how the support evidence reaches the prediction:

Dense correspondence (HSNet). Every query position is correlated with every support position, at every intermediate backbone layer, into a 4D hypercorrelation ${\hat{\mathbf{C}}}_l\in {\mathbb{R}}^{H_l\times W_l\times H_l\times W_l}$, and a stack of 4D convolutions learns to read the geometric patterns in it. Buys: spatial structure that pooling destroys — the multi-level ablation is worth $+4.5$ mIoU over the deepest single level, and removing the fine pyramid levels costs $66.2\to 55.5$. Costs: memory quadratic in feature-map resolution, which is why the input is capped at $400\times 400$ and the finest correlation at $50\times 50$ even after the center-pivot kernel makes the convolutions linear. It also has nothing to say about classification, which is why it cannot be lifted to the FSIS setting.

Prototype matching (the paradigm both HSNet and RefT argue against, and which survives inside RefT’s first reference). Masked average pooling collapses the support foreground to one vector per class. Buys: $O(1)$ support representation, cheap, class-comparable, and it lets RefT process all $N$ classes in one forward pass instead of one pass per class. Costs: all spatial structure. RefT’s Tab. 13 measures the loss directly — GAP 17.0 nAP, RoIAlign 17.2, MaskPool 17.4 — the more accurately the region is extracted, the better, which is the same finding as HSNet’s support-masking ablation in a different currency.

Cross-attention over object queries (RefT). The support branch’s own object queries are filtered to the top-$k$ by mask IoU and cross-attended into the query image’s object queries: $q_Q^{\text{Enhanced}}=\mathrm{softmax}(q_Q\mathrm{TopK}(q_S{)}^{\top })\mathrm{TopK}(q_S)$. Buys: a carrier that entangles where and what, which pooled prototypes cannot. RefT’s Tab. 11 isolates it — the same guidance delivered as prototypes gives 16.3 nAP, delivered as object queries 17.4 — and the probe behind it shows support queries already localize novel objects at bIoU/nIoU $0.84/0.79$ before any novel fine-tuning. Costs: it needs a query-based meta-architecture to exist at all (no RPN detector has object queries to borrow), it costs $-0.5$ bAP by reducing base-class bias, and it degrades at $K=1$, where a single support set does not represent the class distribution — RefT loses to iFS-RCNN at one shot ($5.1$ vs $5.5$ nAP) and only wins from three shots up.

Prompt engineering on a frozen model (PerSAM). Cosine similarity between masked reference features and the test image gives a confidence map $S=\frac{1}{n}{\sum }_i{S}^i$; its argmax and argmin become a positive and a negative point prompt, $S$ additionally re-biases the decoder’s cross-attention as $A^g=\mathrm{softmax}(A+\alpha \mathrm{softmax}(S))$, and the pooled embedding is added to every decoder input token. Buys: zero training, zero risk of overfitting the one shot, immediate transfer to video (DAVIS17 76.1 $\mathcal{J}\&\mathcal{F}$ with no video training) and near-free prompt-type relaxation. Note the correlation itself is the same operation as HSNet’s, pooled rather than fed to a learned 4D encoder — the two papers agree on the primitive and disagree entirely on what to do with it. Costs: the whole method is bounded by what the frozen features already encode. The location prior is purely appearance-based with no notion of instance identity, so near-identical distractors are unresolvable in principle, and the ablation shows cascaded post-refinement ($+11.4$ mIoU) contributes more than every semantic component combined — much of the accuracy is SAM cleaning up after a coarse prompt.

Concept-conditioned detection (SAM3). The prompt (text tokens and/or exemplar tokens) conditions the frame embeddings through a fusion encoder, and 200 object queries decode against them, with recognition factorized out of localization by a global presence token:
$p({\text{query}}_i\text{ matches NP})=p({\text{query}}_i\text{ matches NP}\mid \text{NP appears in image})\cdot p(\text{NP appears in image}).$
Buys: exhaustive multi-instance output, open vocabulary, identities across video, and exemplars that generalize — after 3 exemplars interactive PCS beats even an idealized per-instance PVS correction baseline by $+2.0$ cgF${}_1$, because an exemplar suppresses or detects all similar objects rather than fixing one. Costs: a data engine producing 4M unique noun phrases with AI verifiers, a 750M-parameter backbone, linear inference scaling in the number of tracked objects, and weak generalization to genuinely out-of-domain concepts (aircraft types, medical terms, thermal imagery) — precisely the long tail where the pre-SAM few-shot framing was supposed to help.

The presence head is the mechanism-level statement of this page’s thesis. SAM 3 found that forcing one query to both recognize and localize is counterproductive and factorized them, worth $+1.5$ cgF${}_1$ and IL_MCC $0.77\to 0.82$. The few-shot literature made the opposite mistake at the task level for a decade, bundling recognition and localization into one benchmark number and then optimizing the part that was already solved.

How the foundation-model era changed the problem

The pre-SAM framing is HSNet’s: features are fixed and mediocre (ImageNet, frozen because fine-tuning them overfits), so the research problem is how to generalize from $k$ examples — how to squeeze a usable mask out of a similarity signal computed in a representation that was never trained for the task. Everything follows from that premise. The 4D correlation pyramid exists because the features are too weak to be pooled without loss. The episodic protocol exists to certify generalization to classes the correlation encoder never saw. Even the parameter count (2.6M) is a consequence: with $k$ examples you can afford almost no capacity, so the capacity goes into reading correspondences rather than into features.

The post-SAM framing inverts every one of those premises. SAM segments essentially anything from a point, and its stated limitation is not mask quality but that it “produces masks, not labels” and that it is unclear how to prompt it for semantics. Mask quality is therefore no longer the bottleneck — which is exactly what the FSS-1000 pair above measures: two learnable parameters on frozen SAM features match 2.6M parameters of purpose-built correlation machinery. What remains hard is identification: deciding which of the many masks SAM will happily produce is the one the user meant, and whether the concept is present at all.

Both post-SAM papers are attacks on that residual problem, from opposite ends:

• PerSAM answers “which mask” with feature matching, for a single subject, for free. Its entire mechanism is prompt synthesis — nothing about the mask decoder changes. And its failure mode is an identification failure, not a segmentation failure: scale ambiguity (is the hat part of the teddy bear?) and appearance-identical distractors.
• SAM3 answers “which concept” by training the naming ability in directly, and its own ablations confirm that naming — not masking — is where the gains live: hard negatives per image $0\to 30$ move cgF${}_1$ $28.3\to 43.0$ with IL_MCC $0.44\to 0.68$, and the article notes the gain is “almost entirely image-level recognition”. Adding an AI verifier on top, replacing the presence score, is worth another $+7.2$ cgF${}_1$. None of that touches mask geometry.

RefT is the transitional case and is more interesting for it. It is structurally a pre-SAM paper — base/novel split, episodic fine-tuning, COCO nAP — but its central empirical finding is a post-SAM finding: after base training alone, the model already localizes and segments novel-class objects at nIoU $0.79$, and only the class label is missing; the fine-tuned baseline predicts train as refrigerator while the mask is correct. RefT’s answer is to inject support guidance at the query level where categorization lives. That is the same diagnosis SAM 3 makes with its presence head, reached from inside the old paradigm, one year earlier, without the data to act on it at scale.

The honest summary is that the old framing has not been refuted so much as relocated. Segmenting from $k$ examples is now easy because someone paid for 1.1B masks. Naming from $k$ examples in a domain nobody pretrained on — aircraft variants, thermal imagery, cell types — is still open, and it is the setting where HSNet’s premise (fixed features, learn the correspondence) has not been superseded, because there the foundation model’s features are the weak ones. SAM3’s own stated mitigation for out-of-domain concepts is small-scale fine-tuning or synthetic domain adaptation — both of which require training, which is precisely the thing the foundation-model framing promised to remove.

Which to reach for when

• Segment one specific object across a photo album or a video, one example, no training. PerSAM, and PerSAM-F if the object is hierarchical (a hat on a bear, a teapot on a platform) — the two scale weights are the whole fix and cost ten seconds. If a mask is inconvenient, a box costs about 1 mIoU. Do not reach for a PEFT method here: prompt tuning and Adapter both score below training-free PerSAM.
• Segment every instance of a nameable concept, in a domain a large model plausibly saw. SAM3, with the noun phrase, and add exemplar boxes if text alone is imprecise — three exemplars are worth $+21.6$ cgF${}_1$, and the model is state of the art at 10-shot ODinW13 with no prompt tuning. It also counts (CountBench MAE 0.12) and tracks, which nothing else here does.
• Instance masks plus class labels from a fixed vocabulary, with $\ge 3$ shots, and base classes that must keep working. RefT. Fine-tune CAL, the object queries and the class head, and nothing else — the ablation is unambiguous that fine-tuning more overfits catastrophically. If base training data is unavailable at adaptation time, the CE-BCD loss buys $+6$ to $+8$ bAP for about $1$ nAP.
• At exactly one shot with class labels required. Nothing here is good. RefT explicitly loses to the RPN-based iFS-RCNN at $K=1$ ($5.1$ vs $5.5$ nAP) and says so as a stated limitation.
• A domain the foundation models have not seen — medical, thermal, industrial, microscopy. HSNet is the honest recommendation, and its cross-domain evidence supports it (train COCO-20${}^i$, test PASCAL-5${}^i$: R101 64.1 mIoU 1-shot, beating RePRI’s 63.2 with $18\times$ fewer parameters and no augmentation). The premise it was built on — mediocre frozen features, learn the correspondence — is still the true premise out there. Note also that its correlation is parameter-free and backbone-agnostic, so it composes with a better frozen encoder if you have one.
• You have no examples at all, only a click. SAM, or MobileSAM if it must run on a phone. That is not a few-shot problem and should not be solved with a few-shot method.
• Propagating one annotated frame through a video. SAM2 — but note that PerSAM reaches DAVIS17 $\mathcal{J}\&\mathcal{F}$ 76.1 with no video training, so if your object is stable in appearance the cheap route may suffice.

See also

• HSNet — few-shot semantic segmentation by dense 4D correlation; the pre-foundation-model paradigm at its strongest
• RefT — few-shot instance segmentation on a Mask2Former; the transitional paper that diagnoses classification, not localization, as the bottleneck
• PerSAM — one-shot personalization of frozen SAM, training-free or two parameters
• SAM3 — concept-promptable segmentation; the few-shot problem recast as interactive prompting over a 4M-phrase vocabulary
• SAM — the promptable substrate PerSAM builds on and SAM 3 supersedes
• SAM2 — video propagation from one annotated frame; adjacent, but not few-shot
• DETR Family Comparison — the same cross-paper exercise for the detector line RefT and SAM 3 both descend from
• Your ViT is Secretly an Image Segmentation Model — the argument that the segmentation machinery RefT inherits is largely removable
• Implementing Research Papers — the episodic protocols, unreported $\lambda$ values and split definitions on this page are exactly what decides whether a reproduction lands