Matcher

Matcher is a training-free framework for one-shot (“segment anything with one shot”) segmentation that combines two off-the-shelf vision foundation models, a correspondence extractor and a class-agnostic segmenter, without fine-tuning either.

• Paper: Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang and Chunhua Shen (Zhejiang University / Beijing Academy of Artificial Intelligence), Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching, arXiv:2305.13310.
• Task: given a single reference image with a mask and a target image, locate and segment objects or parts of the same semantics in the target, all without ever training on the downstream task.
• Components: SAM (used purely as a promptable, class-agnostic segmenter) paired with DINOv2 (used purely as a dense feature/correspondence extractor).
• Bridging modules: three purpose-built, training-free modules: bidirectional matching, a robust prompt sampler, and instance-level matching.

Fig. 1: Matcher’s three operations — Correspondence Matrix Extraction (DINOv2 features → patch similarity), Prompts Generation (patch-level matching + robust prompt sampling → SAM prompts), and Controllable Masks Generation (instance-level matching + mask merging → target mask).

Motivation

Large language models solve a wide range of NLP tasks zero/few-shot through in-context learning, without any task-specific architecture or fine-tuning. Vision foundation models (VFMs) have not caught up: CLIP, MAE and DINOv2 provide powerful representations, and SAM provides impressive zero-shot class-agnostic segmentation, but each individually falls short of a full downstream task. SAM in particular can segment any region it is prompted on, but it has no notion of semantics — it cannot tell you which mask corresponds to “the same object as this reference,” and given ambiguous prompts it just returns multiple candidate masks. Matcher’s premise is that combining an all-purpose feature-matching model (semantics, correspondence) with an all-purpose segmentation model (precise boundaries) can jointly cover what neither can do alone — entirely without training, using only a single in-context example.

Architecture

Matcher has no learned parameters of its own. It orchestrates two frozen foundation models:

• DINOv2 (ViT-L/14) as the image encoder, producing dense patch-level features ${\mathbf{z}}_r,{\mathbf{z}}_t\in {\mathbb{R}}^{H\times W\times C}$ for the reference image ${\mathbf{x}}_r$ (with mask $m_r$) and target image ${\mathbf{x}}_t$. DINOv2 is chosen over CLIP/MAE because its joint image- and patch-level self-supervised objective yields much more precise patch-to-patch correspondence (see Ablations below).
• SAM (ViT-H) as the promptable, class-agnostic segmenter, used only to turn point/box prompts into candidate masks — it never sees semantic labels.

The framework runs in three stages, matching the three colored blocks of Fig. 1:

1. Correspondence Matrix Extraction (CME) — compute a dense patch-similarity matrix between reference and target features.
2. Prompts Generation (PG) — turn that matrix into diverse, outlier-robust point/box prompts for SAM via bidirectional matching (Patch-Level Matching) and a Robust Prompt Sampler.
3. Controllable Masks Generation (CMG) — SAM proposes candidate masks from those prompts; Instance-Level Matching scores and selects the high-quality ones, which are then merged into the final target mask.

Key methodology

Correspondence matrix extraction

Given patch features ${\mathbf{z}}_r,{\mathbf{z}}_t\in {\mathbb{R}}^{H\times W\times C}$, the correspondence matrix $\mathbf{S}\in {\mathbb{R}}^{HW\times HW}$ is simply cosine similarity between all pairs of patches:

$$(\mathbf{S}{)}_{ij}=\frac{{\mathbf{z}}_r^i\cdot {\mathbf{z}}_t^j}{\Vert {\mathbf{z}}_r^i\Vert \cdot \Vert {\mathbf{z}}_t^j\Vert },\mathbf{S}=\mathrm{sim}({\mathbf{z}}_r,{\mathbf{z}}_t)$$

Naively taking the arg-max matches per reference patch gives a coarse, fragmented mask riddled with outliers — the encoder produces wrong matches whenever context is ambiguous or multiple instances are present.

Bidirectional matching (Patch-Level Matching)

Fig. 2: Bidirectional matching in three steps. Purple points are the matched points that survive filtering; red points are outliers removed by the reverse pass.

To suppress these outliers, Matcher performs a three-step bidirectional matching procedure between the points $P_r=\{{\mathbf{p}}_r^i{\}}_{i=1}^L$ sampled on the reference mask $m_r$ and the target features ${\mathbf{z}}_t$:

1. Forward matching: bipartite (Hungarian) matching between $P_r$ and ${\mathbf{z}}_t$ using ${\mathbf{S}}^{\to }=\mathrm{sim}(P_r,{\mathbf{z}}_t)$ gives forward matched points $P_t^{\to }=\{{\mathbf{p}}_t^i{\}}_{i=1}^L$ on the target image.
2. Reverse matching: bipartite matching back from $P_t^{\to }$ to ${\mathbf{z}}_r$ using ${\mathbf{S}}^{\leftarrow }=\mathrm{sim}({\mathbf{z}}_r,P_t^{\to })$ gives reverse matched points $P_r^{\leftarrow }=\{{\mathbf{p}}_r^i{\}}_{i=1}^L$ back on the reference image.
3. Mask filtering: keep only forward matches whose reverse partner actually lands back inside the reference mask:

$$\hat{P}=\{{\mathbf{p}}_t^i\in P_t^{\to }\mid {\mathbf{p}}_r^i\text{ in }{m}_r\}$$

This cycle-consistency check is cheap and highly effective: the ablations show bidirectional matching improves over forward-only matching by +2.1% mIoU on COCO-20${}^i$, +5.9% on FSS-1000, and +6.0% $J\&F$ on DAVIS 2017, while reverse-only matching (no forward guidance) collapses performance entirely.

Robust Prompt Sampler (RPS)

The filtered matched points $\hat{P}$ are clustered into $K$ groups with k-means++ into clusters ${\hat{P}}_k$, then three types of prompt subsets are sampled to cover different semantic granularities:

• Part-level prompts: sampled within each cluster $P^p\subset {\hat{P}}_k$.
• Instance-level prompts: sampled across all matched points $P^i\subset \hat{P}$.
• Global prompts: sampled from the cluster centers $P^g\subset C=\{c_1,\dots ,c_k\}$, to encourage full coverage.

Sampling multiple diverse prompt groups (rather than one point per matched cluster) both increases mask-proposal diversity and suppresses fragmented false positives caused by residual matching outliers. Each prompt group is fed to SAM as points (or the smallest axis-aligned box around the matched points, for part segmentation) to produce a mask proposal.

Instance-level matching (Controllable Masks Generation)

SAM’s proposals still include false-positive masks caused by indistinguishable edge/background features. Matcher filters and ranks them by formulating relevance to the reference mask as an Optimal Transport (OT) problem between the dense semantic features inside the reference mask and inside each mask proposal, using the Earth Mover’s Distance (EMD):

$$\underset{\pi }{\min }\sum _{i=1}^m\sum _{j=1}^n{c}_{ij}{\pi }_{ij}\text{s.t.}\sum _{j=1}^n{\pi }_{ij}=u_i,\sum _{i=1}^m{\pi }_{ij}=d_j,\sum _i{u}_i=\sum _j{d}_j,{\pi }_{ij}\ge 0$$

with the cost matrix $\mathbf{C}=\frac{1}{2}(1-\mathbf{S})$, uniform supplies/demands ($u_i=1/m$, $d_j=1/n$), suppliers being the $m$ reference patches inside $m_r$ and demanders the $n$ target patches inside a mask proposal $m_p$; the resulting distance is denoted $emd$.

Two further metrics assess proposal quality directly from the matched points ${\hat{P}}_{mp}=\{{\mathbf{p}}_t^i\in P_t^{\to }\mid {\mathbf{p}}_t^i\text{ in }{m}_p\}$:

$$purity=\frac{Num({\hat{P}}_{mp})}{Area(m_p)},coverage=\frac{Num({\hat{P}}_{mp})}{Num(\hat{P})}$$

A high purity favors small, precise (part- or single-instance) masks; a high coverage favors large masks that capture whole or multiple instances (Fig. 5). These are combined into a single selection score:

$$score=\alpha \cdot (1-emd)+\beta \cdot purity\cdot coverag{e}^{\lambda }$$

where $\alpha ,\beta ,\lambda$ are dataset-dependent regularization coefficients (see Hyperparameters). Mask proposals are filtered by $emd$/$purity$/$coverage$ thresholds, then the top-$k$ by score are kept and merged (Controllable Masks Merging) — by controlling $k$, Matcher can output a controllable number of instances of the same semantics in the target image, e.g. segmenting either one or all matching instances on request.

Fig. 5: Higher purity selects tighter part/single-instance masks; higher coverage selects broader whole/multi-instance masks.

Video object segmentation extension

For VOS, Matcher maintains a reference memory of features and intermediate predictions from previous frames, selecting which frames to retain according to their score (from instance-level matching), with a time-decay ratio so that nearby frames are weighted more heavily. This lets a purely one-shot, training-free image framework track objects across a video without any video-specific training.

Hyperparameters

As reported in the paper’s Appendix B (Implementation Details), all without any training of Matcher itself:

Setting	Value
Image encoder	DINOv2, ViT-L/14
Segmenter	SAM, ViT-H
Input size (semantic / part segmentation)	$518\times 518$
Input size (video object segmentation)	$896\times 504$
Number of k-means++ clusters $K$	8
COCO-20${}^i$ / LVIS-92${}^i$: $emd$ threshold	0.67
COCO-20${}^i$ / LVIS-92${}^i$: $purity$ threshold	0.02
COCO-20${}^i$ / LVIS-92${}^i$: $\alpha ,\beta ,\lambda$	1.0, 0.0, 0.0
FSS-1000: $\alpha ,\beta ,\lambda$	0.8, 0.2, 1.0
FSS-1000 prompts	sampled from global (cluster-center) prompts
PASCAL-Part / PACO-Part: $coverage$ threshold	0.3
PASCAL-Part / PACO-Part: $\alpha ,\beta ,\lambda$	0.5, 0.5, 0.0
PASCAL-Part / PACO-Part prompts	smallest axis-aligned box over matched points
VOS: $emd$ threshold	0.75
VOS: $\alpha ,\beta ,\lambda$	0.4, 1.0, 1.0
VOS: number of retained memory frames	4 (optimum; tested 1/2/4/6)
VOS prompts	sampled from global (cluster-center) prompts

Results

Matcher is evaluated purely zero-shot/training-free against both specialist few-shot-segmentation models (trained in-domain, e.g. HSNet, VAT, FPTrans, MSANet) and generalist models (Painter, SegGPT, PerSAM/PerSAM-F).

One-shot semantic segmentation (mean mIoU): 52.7% one-shot / 60.7% few-shot on COCO-20${}^i$ (surpassing specialist SOTA MSANet’s 51.1% and matching generalist SegGPT, whose training data includes COCO), 87.0% / 89.6% on FSS-1000, and 33.0% / 40.0% on the paper’s newly proposed LVIS-92${}^i$ benchmark — a harder 920-class, cross-dataset benchmark built from LVIS — beating generalist SOTA SegGPT by +14.4%/+14.6%. Against the concurrent training-free PerSAM: +29.2% mIoU on COCO-20${}^i$, +11.4% on FSS-1000, +10.7% on LVIS-92${}^i$, underscoring that SAM alone (as PerSAM relies on) lacks the semantic understanding needed for this task.

One-shot object part segmentation, evaluated on two new benchmarks built by the authors (PASCAL-Part: 56 parts across animals/indoor/person/vehicles; PACO-Part: 303 object parts derived from PACO): Matcher reaches 42.9% mean mIoU on PASCAL-Part and 34.7% mean F-score on PACO-Part, beating all compared methods (including PerSAM and SegGPT) by roughly 10 points mean mIoU on both.

Video object segmentation (DAVIS 2017 val / DAVIS 2016 val, semi-supervised, $J\&F$): 79.5% / 86.1%, competitive with methods trained on video data (e.g. XMem: 87.7%/92.0%) and clearly ahead of other training-free/no-video-data baselines SegGPT (75.6%/83.7%) and PerSAM-F (71.9%/–).

Ablations confirm each component’s necessity: removing instance-level matching (ILM) collapses COCO-20${}^i$ mIoU from 52.7% to 29.0%; forward-only (non-bidirectional) matching drops COCO-20${}^i$ by 2.1 points and FSS-1000 by 5.9 points; DINOv2 clearly outperforms CLIP and MAE as the encoder (52.7 vs. 32.2 vs. 18.8 mIoU on COCO-20${}^i$); and an upper-bound analysis (oracle mask selection) shows substantial headroom remains (e.g. 83.6% vs. 52.7% on COCO-20${}^i$), meaning most of the error budget sits in mask selection, not mask proposal.

Fig. 3a: Matcher produces cleaner object and part masks than SegGPT and PerSAM-F on both familiar and cross-style (e.g. toy, sketch) images.

Limitations

The paper’s stated limitation is that Matcher’s instance-level matching inherits the limitations of the underlying image encoder, which restrains its effectiveness specifically for instance-level segmentation (as opposed to semantic/part-level). The authors note the framework’s competitive VOS performance and its controllable multi-instance output as evidence it nonetheless has headroom for instance segmentation, left to future work. More generally, being fully training-free and modular, Matcher’s ceiling is bounded by the quality of its frozen backbones — the upper-bound ablation shows a large residual gap versus oracle mask selection, and dependence on Hungarian bipartite matching / OT solving adds computational cost compared to end-to-end trained models.

See also

• SAM — the class-agnostic promptable segmenter Matcher wraps, unmodified and untrained, purely as a mask proposal engine.
• HSNet — a trained specialist few-shot segmentation model Matcher is benchmarked against on COCO-20${}^i$/FSS-1000/LVIS-92${}^i$.
• PerSAM — the concurrent training-free (and lightly-tuned PerSAM-F) approach to adapting SAM for personalized/one-shot segmentation that Matcher substantially outperforms, illustrating the value of pairing SAM with a dedicated correspondence model rather than using SAM’s own features.
• RefT — another few-shot/reference-guided segmentation approach relevant to the one-shot segmentation landscape Matcher operates in.