PANet

PANet (Prototype Alignment Network) is a metric-learning approach to few-shot semantic segmentation.

• Paper: Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng (NUS), PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment, ICCV 2019, arXiv:1908.06391.
• Core idea: embed support and query images into a shared feature space, derive one prototype vector per class from the support set by masked average pooling, and segment the query by assigning each pixel to its nearest prototype under cosine distance.
• Classifier: non-parametric, with no learnable segmentation head.
• Main contribution: prototype alignment regularization (PAR), a training-time regularizer that reverses the roles of support and query, using the query image and its predicted mask as a new “support set” to segment the original support images, so that support information is exploited beyond mere masking.

Overview of PANet. Support and query images are mapped into embedding features (circles and triangles). Prototypes (solid circles) are learned per class from the support set, and the query is segmented by nearest-prototype matching (dashed lines). PAR performs the reverse segmentation, support from query, during training only.

Motivation

Prior few-shot segmentation methods (e.g. Shaban et al.’s conditioning branch, Rakelly et al.’s feature concatenation, Zhang et al.’s SG-One) all funnel support information through a parametric module that fuses support and query features and predicts the segmentation directly. PANet identifies two problems with this scheme. First, it does not separate knowledge extraction from the segmentation process — the model representation ends up entangled with the semantic content of the support set, which hurts generalization to unseen classes. Second, these methods use the support annotations only to mask out background before pooling; they never use the support masks as a learning target, so a large source of supervisory signal goes unused.

PANet addresses the first problem by explicitly separating the pipeline into prototype extraction (a representation-learning step) and non-parametric metric learning (a matching step with zero extra parameters). It addresses the second with PAR, which forces support masks to also act as ground truth in a reverse-direction segmentation task, so the model must learn embeddings for which the query’s predicted prototypes segment the support well too.

Problem setup

The task follows the standard episodic few-shot segmentation protocol (as in HSNet): training and test class sets ${\mathcal{C}}_{\text{seen}},{\mathcal{C}}_{\text{unseen}}$ are disjoint. A $C$-way $K$-shot episode $({\mathcal{S}}_i,{\mathcal{Q}}_i)$ provides a support set ${\mathcal{S}}_i=\{(I_{c,k},M_{c,k})\}$ with $K$ image/mask pairs for each of $C$ classes, and a query set ${\mathcal{Q}}_i$ of unlabeled images from the same classes; the model predicts the query masks having seen only ${\mathcal{S}}_i$, with no further optimization at test time.

Architecture and prototype learning

A single shared feature extractor (VGG-16, first 5 conv blocks; maxpool4 stride set to 1 and conv5 convolutions replaced by dilated convolutions with dilation 2, to preserve spatial resolution and enlarge the receptive field, following the FCN convention) embeds both support and query images into dense feature maps $F_{c,k}$.

Class prototypes are computed by masked average pooling with late fusion — the support images are first embedded normally, and the class mask is only applied to pool the resulting feature map (rather than masking the input image before embedding, “early fusion”, which PANet argues is worse for keeping the input distribution consistent for the shared extractor):

$p_c=\frac{1}{K}{\sum }_k\frac{{\sum }_{x,y}{F}_{c,k}^{(x,y)}1[M_{c,k}^{(x,y)}=c]}{{\sum }_{x,y}1[M_{c,k}^{(x,y)}=c]},$

where $(x,y)$ indexes spatial locations. A background prototype is computed analogously, pooling over all locations of all support images that do not belong to any of the $C$ episode classes:

$p_{\text{bg}}=\frac{1}{CK}{\sum }_{c,k}\frac{{\sum }_{x,y}{F}_{c,k}^{(x,y)}1[M_{c,k}^{(x,y)}\notin {\mathcal{C}}_i]}{{\sum }_{x,y}1[M_{c,k}^{(x,y)}\notin {\mathcal{C}}_i]}.$

Non-parametric metric learning

Given the prototype set $\mathcal{P}=\{p_c\mid c\in {\mathcal{C}}_i\}\cup \{p_{\text{bg}}\}$ and the query feature map $F_q$, a softmax over negative distances to every prototype gives a per-pixel probability map:

${\tilde{M}}_{q;j}^{(x,y)}=\frac{\exp (-\alpha d(F_q^{(x,y)},p_j))}{{\sum }_{p_{j^{\prime }}\in \mathcal{P}}\exp (-\alpha d(F_q^{(x,y)},p_{j^{\prime }}))},{\hat{M}}_q^{(x,y)}=\arg {\max }_j{\tilde{M}}_{q;j}^{(x,y)}.$

The distance $d$ is cosine distance multiplied by a fixed scale $\alpha =20$, rather than squared Euclidean distance as in the original Prototypical Networks. The paper found cosine distance more stable and easier to optimize because it is bounded, and that scaling it by $\alpha$ recovers the performance advantage previously attributed to squared Euclidean distance. This matching step introduces no learnable parameters — segmentation is a direct nearest-prototype lookup, unlike HSNet’s later 4D-convolutional encoder-decoder or SG-One’s learned similarity module.

The segmentation loss is cross-entropy against the query ground truth over all $N$ pixel locations:

${\mathcal{L}}_{\text{seg}}=-\frac{1}{N}{\sum }_{x,y}{\sum }_{p_j\in \mathcal{P}}1[M_q^{(x,y)}=j]\log {\tilde{M}}_{q;j}^{(x,y)}.$

Prototype alignment regularization (PAR)

Illustration of the pipeline in a 2-way 1-shot example. Block (a): the standard support-to-query segmentation, with prototypes from masked average pooling matched to query features by cosine distance. Block (b): the proposed PAR, which treats the query image and its predicted mask as a new support set and segments the original support images, computing ${\mathcal{L}}_{\text{PAR}}$ against the support ground truth.

Without PAR, information flows only one way — from support to query — and the support masks are used solely to mask, never as a target to predict. PAR closes the loop: after predicting the query mask ${\hat{M}}_q$, the query image and its predicted mask are treated as a new support set, from which a second set of prototypes $\bar{\mathcal{P}}=\{{\bar{p}}_c\mid c\in {\mathcal{C}}_i\}\cup \{{\bar{p}}_{\text{bg}}\}$ is extracted (same Eqns. for $p_c,p_{\text{bg}}$, just applied to the query). These prototypes are then used to segment the original support images:

${\tilde{M}}_{c,k;j}^{(x,y)}=\frac{\exp (-\alpha d(F_{c,k}^{(x,y)},{\bar{p}}_j))}{{\sum }_{{\bar{p}}_{j^{\prime }}\in \{{\bar{p}}_c,{\bar{p}}_{\text{bg}}\}}\exp (-\alpha d(F_{c,k}^{(x,y)},{\bar{p}}_{j^{\prime }}))},$

with a loss against the known support ground truth:

${\mathcal{L}}_{\text{PAR}}=-\frac{1}{CKN}{\sum }_{c,k,x,y}{\sum }_{p_j\in \mathcal{P}}1[M_{c,k}^{(x,y)}=j]\log {\tilde{M}}_{c,k;j}^{(x,y)}.$

The total training loss is $\mathcal{L}={\mathcal{L}}_{\text{seg}}+\lambda {\mathcal{L}}_{\text{PAR}}$, with $\lambda =1$ found to work as well as any other tested value. PAR is applied only during training (on ${\mathcal{D}}_{\text{train}}$) — at test time inference is unchanged from block (a), so PAR costs nothing extra at inference. Because the reverse-direction segmentation reuses the exact same prototype-extraction and matching machinery, PAR introduces zero additional learnable parameters; it only shapes the embedding space that VGG-16 is trained to produce.

The intuition: if the query’s predicted mask is used to build prototypes that can also segment the support well, the query and support prototypes must be mutually consistent in the embedding space. Empirically, PAR measurably reduces the average Euclidean distance between query-derived and support-derived prototypes (32.2 vs. 42.6 without PAR, over 1,000 sampled 5-shot episodes on PASCAL-5${}^i$ split-1), speeds up training convergence (lower training loss at every iteration checkpoint, Fig. 5), and improves final mIoU by 0.9 points (1-shot) and 0.8 points (5-shot).

Generalization to weaker annotations

Because prototypes are extracted by simple masked pooling rather than a parametric module tuned to dense masks, PANet is directly applicable to weaker support annotations — scribbles and bounding boxes — generated automatically from the dense masks at test time, with no retraining. This works because late fusion means annotations only ever gate which spatial locations get pooled; a sparser or noisier gate still yields a usable prototype.

Hyperparameters

Setting	Value
Feature extractor	VGG-16, first 5 conv blocks, ImageNet (ILSVRC)-pretrained
Architectural modification	maxpool4 stride set to 1; conv5 convolutions replaced by dilated convolutions, dilation 2
Input resolution	$417\times 417$
Data augmentation	random horizontal flipping only
Distance function	cosine distance, scaled by $\alpha =20$
PAR weight	$\lambda =1$
Optimizer	SGD, momentum 0.9
Training iterations	30,000
Learning rate	initialized at $1\times 10^{-3}$, ×0.1 every 10,000 iterations
Weight decay	0.0005
Batch size	1
$N_{\text{query}}$	1 (one query image per episode)
Evaluation	4-fold cross-validation on PASCAL-5${}^i$ / COCO-20${}^i$; results averaged over 5 runs of 1,000 episodes each with different random seeds

Results

PASCAL-5${}^i$, mean-IoU averaged over the 4 folds:

Method	1-shot mIoU	5-shot mIoU	Params
OSLSM	40.8	43.9	272.6M
co-FCN	41.1	41.4	34.2M
SG-One	46.3	47.1	19.0M
PANet-init (untrained baseline)	35.3	46.7	14.7M
PANet	48.1	55.7	14.7M

PANet outperforms the prior state of the art by 1.8 points (1-shot) and 8.6 points (5-shot) mIoU, using fewer parameters and no decoder or post-processing. Notably, the gap between 1-shot and 5-shot is much larger for PANet (7.6 points) than for competing methods (< 3.1 points), meaning it exploits additional support examples far more effectively — consistent with PAR’s aim of extracting richer information from the support set. Even the untrained PANet-init baseline (ImageNet weights only) is competitive with the trained state of the art in the 5-shot setting.

Binary-IoU on PASCAL-5${}^i$: PANet reaches 66.5 / 70.7 (1-shot / 5-shot), also state of the art. In 2-way multi-class evaluation, PANet outperforms prior work by more than 20 points in both metrics.

MS COCO-20${}^i$: PANet outperforms A-MCG by 7.2 points (1-shot) and 8.2 points (5-shot) mean-IoU.

Weak annotations (dense mask 48.1 / 55.7 mIoU as reference): scribbles reach 44.8 / 54.6, bounding boxes reach 45.1 / 52.8 (1-shot / 5-shot). Scribbles outperform bounding boxes more in the 5-shot setting (2 points), attributed to bounding boxes introducing more noise into the prototype while scribbles give more representative pooling regions even with sparser coverage.

Limitations

• No decoder or refinement. Predictions come directly from per-pixel nearest-prototype assignment with no learned upsampling/refinement head, which the paper notes produces occasional “unnatural patches” — inconsistent per-location predictions that a decoder or CRF post-processing could smooth out.
• Single global prototype per class. Masked average pooling collapses all spatial structure of the support foreground into one vector, discarding intra-class part or pose variation — later work (e.g. HSNet) argues this is precisely why prototype methods plateau, motivating dense correspondence approaches that skip pooling altogether.
• Confusable classes. The paper reports a specific failure mode where visually and structurally similar classes (chairs vs. tables) end up with overlapping prototypes in the embedding space and are not reliably distinguished.
• PAR is a training-only regularizer, not a mechanism the model can use to correct itself at test time; test-time predictions still depend entirely on the quality of the single forward support-to-query pass.
• $K$-shot aggregation is at the prototype level (averaging $K$ support images into one prototype via Eqn. 1), simpler than HSNet’s per-shot voting, but this means all $K$ shots are pooled into a single fixed representation before any query interaction happens.

See also

• HSNet — explicitly positions itself against the prototypical paradigm PANet represents: instead of collapsing the support into a global vector, it keeps a dense multi-layer correlation between every support and query location, motivated by exactly the spatial-structure loss PANet’s masked average pooling incurs.
• RefT — a later few-shot segmentation approach; useful point of comparison for how the field’s matching mechanisms evolved past simple prototype-cosine matching.
• PerSAM — shares PANet’s core mechanism of masked-average-pooling a reference embedding and matching by cosine similarity to locate the target, but training-free and built on a frozen SAM rather than an end-to-end trained backbone.
• Implementing Research Papers — PANet’s episodic 4-fold cross-validation protocol and averaging over 5 runs of 1,000 episodes is the same kind of evaluation-protocol detail worth matching exactly when reproducing few-shot segmentation results.