PerSAM

PerSAM (Personalize Segment Anything Model) customizes SAM to a specific visual concept (your dog, your backpack, your teapot) from one image plus one mask.

• Paper: Zhang, Jiang, Guo, Yan, Pan, Ma, Dong, Qiao, Gao, Li (CUHK MMLab / Shanghai AI Lab), Personalize Segment Anything Model with One Shot, May 2023, ICLR 2024, arXiv:2305.03048.
• Core idea: after the one-shot customization, the model segments that concept automatically in new images and videos without any further human prompting.
• Training: the base method is entirely training-free.
• PerSAM-F: a fine-tuned variant that tunes exactly 2 parameters in 10 seconds on one A100 to resolve mask-scale ambiguity.

Motivation

SAM is promptable but not personal. Given a photo album, cropping out your pet dog with vanilla SAM means finding the dog in every image by hand and clicking it — labour-intensive and defeating the point of a foundation model. The paper defines the task of personalized object segmentation: the user supplies a single reference image $I_R$ and a mask $M_R$ (an accurate segmentation or a rough on-the-fly sketch — or, per the ablation, just a bounding box), and the model must then find and segment that same instance in arbitrary new images or video frames, unprompted.

The design constraint is that SAM’s pre-trained weights are worth preserving: they encode 1.1B masks of general segmentation ability. So PerSAM touches no weights at all, and PerSAM-F touches two scalars.

Recap: SAM’s interfaces

SAM is an image encoder ${\mathrm{Enc}}_I$, a prompt encoder ${\mathrm{Enc}}_P$, and a lightweight mask decoder ${\mathrm{Dec}}_M$:

$F_I={\mathrm{Enc}}_I(I),T_P={\mathrm{Enc}}_P(P),$

with $F_I\in {\mathbb{R}}^{h\times w\times c}$ and $T_P\in {\mathbb{R}}^{k\times c}$ for $k$ prompts. Learnable mask tokens $T_M$ are concatenated as prefixes to the prompt tokens and decoded against the image feature:

$M={\mathrm{Dec}}_M(F_I,\mathrm{Concat}(T_M,T_P)).$

Every PerSAM technique attaches to one of these three interfaces: it synthesises $T_P$ from the one-shot data, modulates the attention inside ${\mathrm{Dec}}_M$, and augments the input tokens with a semantic embedding.

Training-free PerSAM

Location confidence map

Encode both the reference and the test image with the same (frozen) encoder — SAM’s own backbone by default, though any pre-trained vision encoder works:

$F_I={\mathrm{Enc}}_I(I),F_R={\mathrm{Enc}}_I(I_R).$

Crop the foreground of the reference feature map with the mask to get $n$ local features, one per foreground pixel:

$\{T_R^i{\}}_{i=1}^n=M_R\circ F_R,T_R^i\in {\mathbb{R}}^{1\times c},$

where $\circ$ is spatial-wise multiplication. Each local feature is matched against the whole test image by cosine similarity (both sides are pixel-wise L2-normalized), giving $n$ local confidence maps:

$\{S^i{\}}_{i=1}^n=\{F_I{T_R^i}^{\mathsf{T}}{\}}_{i=1}^n,S^i\in {\mathbb{R}}^{h\times w}.$

Each $S^i$ localizes a different part of the concept — the head, the body, a paw. Averaging them yields the overall confidence map:

$S=\frac{1}{n}{\sum }_{i=1}^n{S}^i\in {\mathbb{R}}^{h\times w}.$

Because it aggregates the appearance of every foreground pixel rather than a single global embedding, $S$ is robust to pose and viewpoint change.

Positive-negative location prior

The pixels with the highest and lowest confidence in $S$, $P_h$ and $P_l$, are taken as a positive and a negative point prompt:

$T_P={\mathrm{Enc}}_P(P_h,P_l)\in {\mathbb{R}}^{2\times c}.$

SAM then segments the contiguous region around the positive point while discarding the region around the negative one. This alone is the whole “automatic prompting” mechanism — no detector, no training.

Target-guided attention

Two points are a thin signal, so PerSAM also injects $S$ inside the decoder. Every token-to-image cross-attention map $A\in {\mathbb{R}}^{h\times w}$ (post-softmax) is re-biased by the confidence map:

$A^g=\mathrm{softmax}(A+\alpha \cdot \mathrm{softmax}(S)),$

with $\alpha$ a balancing factor. This compels mask and prompt tokens to aggregate features from the foreground target rather than from unimportant background.

Target-semantic prompting

Vanilla SAM prompts carry only low-level positional information (a coordinate, a box). PerSAM adds a high-level semantic cue: average-pool the local features into a global target embedding

$T_R=\frac{1}{n}{\sum }_{i=1}^n{T}_R^i\in {\mathbb{R}}^{1\times c},$

and add it element-wise to every decoder input token:

$T^g=\mathrm{Repeat}(T_R)+\mathrm{Concat}(T_M,T_P).$

Cascaded post-refinement

The initial mask can have rough edges and isolated background blobs. PerSAM runs the decoder twice more: step one re-prompts with the current mask plus the original point pair; step two additionally supplies the bounding box enclosing that mask. Only the lightweight decoder is iterated (the image encoder runs once), so this costs about 2% extra latency.

PerSAM-F: scale-aware fine-tuning

The remaining failure mode is mask-scale ambiguity for hierarchically structured objects: for a teapot on a platform, a positive point on the body with a negative point that fails to exclude a similarly-coloured platform can make PerSAM segment the wrong granularity. SAM’s own answer is to emit three masks (whole / part / subpart) and let a human choose — unacceptable for an automatic pipeline.

PerSAM-F keeps SAM completely frozen, takes SAM’s three-scale outputs $M_1,M_2,M_3$, and learns a weighted summation:

$M=w_1\cdot M_1+w_2\cdot M_2+(1-w_1-w_2)\cdot M_3,$

with $w_1,w_2$ both initialized to $1/3$. They are fitted by one-shot fine-tuning on the reference image, treating the provided mask as ground truth. Two scalars, 10 seconds, one A100 — small enough that overfitting the one-shot sample is not a concern, which the PEFT ablation below confirms is not true of the usual alternatives.

PerSAM-assisted DreamBooth

DreamBooth fine-tunes a diffusion model on 3–5 photos of a subject and binds it to an identifier [V], computing the reconstruction loss over the entire image. The background therefore leaks into [V]: prompting “a [V] cat on a beach” can still reproduce the stairs or forest from the training photos. PerSAM segments the foreground subject in each training image, and the gradient is blocked for background pixels, so Stable Diffusion memorizes only the target’s appearance. The result is better subject fidelity and more diverse, prompt-faithful backgrounds.

PerSeg and results

For evaluation the authors annotate PerSeg, a personalized-segmentation dataset built from the raw images of subject-driven diffusion work (DreamBooth, Textual Inversion, Custom Diffusion): 40 objects — daily necessities, animals, buildings — each with 5–7 images and masks in different poses or scenes, with one image-mask pair fixed as the one-shot input. Metrics are mIoU and bIoU.

Method	mIoU	bIoU	Learnable params
Painter	56.4	42.0	354M
VP (Visual Prompting)	65.9	25.5	383M
SEEM*	87.1	55.7	341M
SegGPT*	94.3	76.5	354M
PerSAM	89.3	71.7	0
PerSAM-F	95.3	77.9	2

(* concurrent work.) Training-free PerSAM already beats Painter, VP and SEEM; the 2-parameter PerSAM-F edges past SegGPT by +2.4 mIoU / +4.1 bIoU at a fraction of the compute.

On video object segmentation (DAVIS 2017 val, first frame given), with no video training at all:

Method	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Painter	34.6	28.5	40.8
SEEM	58.9	55.0	62.8
SegGPT	75.6	72.5	78.6
PerSAM	66.9	71.3	75.1
PerSAM-F	76.1	74.9	79.7

PerSAM beats Painter by +32.3 $\mathcal{J}\&\mathcal{F}$, and PerSAM-F outperforms methods fully trained on video data. On one-shot semantic and part segmentation, PerSAM-F reaches 86.3 mIoU on FSS-1000, 18.4 on LVIS-92${}^i$, 32.9 on PASCAL-Part and 22.7 on PACO-Part — consistently above Painter and comparable to SegGPT, showing the method is not restricted to object-level personalization.

Ablations

Components, cumulative on PerSeg mIoU:

Variant	mIoU	Gain
Positive prior only	69.1	—
+ negative prior	72.5	+3.4
+ cascaded post-refinement	83.9	+11.4
+ target-guided attention	85.8	+1.9
+ target-semantic prompting	89.3	+3.5
+ scale tuning (PerSAM-F)	95.3	+6.0

Cascaded post-refinement is by far the largest single contributor; scale tuning is second.

Against other PEFT methods (all freezing SAM, tuning only injected modules in the decoder):

Method	Params	mIoU
PerSAM (training-free)	0	89.32
Prompt tuning	12K	76.5
Adapter	196K	78.3
LoRA	293K	90.0
3 mask weights	3	92.9
PerSAM-F (2 weights)	2	95.3

Prompt tuning and Adapter overfit the one-shot sample and fall below the training-free baseline. Even LoRA barely improves on it. Fewer parameters win here, and constraining the weights to sum to one (2 params) beats three free weights.

Relaxing the input from a mask to a bounding box (using off-the-shelf SAM to turn the box into the one-shot mask) costs PerSAM only 89.3 → 88.1 mIoU and PerSAM-F 95.3 → 94.9, whereas the same relaxation devastates the baselines (SegGPT 94.3 → 36.0).

Hyperparameters

The paper reports a deliberately minimal training configuration — there is essentially nothing to tune:

Setting	Value
Trainable parameters (PerSAM)	0 (training-free)
Trainable parameters (PerSAM-F)	2 — the mask weights $w_1,w_2$
Initialization of $w_1,w_2$	$1/3$ each
Fine-tuning data	the single reference image; its given mask as ground truth
Fine-tuning time	10 seconds on a single A100
Backbone state	entire SAM frozen
Image encoder	SAM’s frozen ${\mathrm{Enc}}_I$ by default (any pre-trained vision encoder is admissible)
Attention modulation factor	$\alpha$ (balancing factor in Eq. 8; value not stated)
Cascaded post-refinement	2 steps (mask prompt, then mask + box prompt), ~2% extra latency
PerSeg	40 objects, 5–7 image-mask pairs each
DreamBooth assistance	all DreamBooth hyperparameters kept unchanged; Stable Diffusion as base model

Limitations

• Scale ambiguity persists in the training-free model. Hierarchical objects (a hat on a teddy bear, the head of a robot toy) can be segmented at the wrong granularity, since both part and whole are valid SAM masks. PerSAM-F only alleviates this, by picking a fixed blend of the three scales per concept.
• The location prior is purely appearance-based. Cosine similarity between local features has no notion of instance identity beyond appearance, so distinguishing one object among several near-identical ones relies on the same feature statistics that make them look alike.
• One negative point is a coarse background signal; the teapot failure case is precisely a negative prior that lands on a same-coloured distractor.
• Reference quality bounds everything. The one-shot mask defines the local feature bank; a box-derived mask already costs about a point of mIoU.
• Weights are per-concept. PerSAM-F’s $w_1,w_2$ are fitted for a particular subject and are not a general-purpose scale selector.

See also

• SAM — the frozen model PerSAM personalizes; its three-scale mask output is what PerSAM-F learns to blend.
• SAM2 — video segmentation with an explicit memory bank, an alternative to PerSAM’s training-free per-frame matching.
• SAM3 — prompting SAM with a concept (noun phrase or exemplar) rather than a coordinate; the same “make SAM find it itself” goal, solved by training rather than by feature matching.
• MobileSAM — the complementary axis: keep SAM’s behaviour, shrink its encoder.