YOLOE

YOLOE (“YOLO, real-time seeing anything”) is a single YOLO-based model that does open-vocabulary object detection and instance segmentation at closed-set YOLO speed.

• Paper: Wang, Liu, Chen, Lin, Han, Ding (Tsinghua University), YOLOE: Real-Time Seeing Anything, 2025, arXiv:2503.07465.
• Prompt types: text, visual (boxes/masks), and no prompt.
• Modules: each prompt mode gets one lightweight, purpose-built module: RepRTA (text), SAVPE (visual), LRPC (prompt-free).
• Base: built on the YOLOv8 architecture (also validated on YOLO11).
• Headline result: beats YOLO-Worldv2 on LVIS with 3× less training cost and up to 1.4× faster inference, and transfers to COCO better than the closed-set YOLOv8 it is fine-tuned from.

Motivation

Closed-set YOLOs (YOLOv8, YOLOv10) are fast but fixed to a predefined label set. Open-prompt systems exist but each pays a different price:

• Text prompts (GLIP, Grounding DINO, YOLO-World): cross-modality fusion between text and image features is expensive, and cost grows with vocabulary size.
• Visual prompts (T-Rex2, DINOv): transformer-heavy prompt encoders or an extra visual backbone, hard to deploy on edge hardware.
• Prompt-free (GenerateU, GRiT, DINO-X): a language model generates category names per region — hundreds of millions of parameters.
• Unified models like DINO-X cover all three but with heavy training and inference cost.

YOLOE’s claim is that all three can live in one YOLO, with zero inference overhead for text prompts and no language model for prompt-free.

Architecture

Figure 2 (paper): YOLOE overview — shared backbone/PAN/heads, plus RepRTA (text), SAVPE (visual) and LRPC (prompt-free).

The base is the standard YOLO layout: backbone → PAN producing $P_3,P_4,P_5$ → per-anchor-point heads for box regression, mask (prototypes + mask coefficients, as in YOLACT / YOLOv8-seg) and an object embedding head. The embedding head is the classification head with its last $1\times 1$ conv’s output channels changed from the class count to an embedding dimension $D$.

Classification is a contrast between anchor-point embeddings and prompt embeddings:

$\text{Label}=\mathcal{O}\cdot {\mathcal{P}}^T:{\mathbb{R}}^{N\times D}\times {\mathbb{R}}^{D\times C}\to {\mathbb{R}}^{N\times C}$

with $N$ anchor points, $C$ prompts, $D$ embedding dim. $\mathcal{O}$ is the object embeddings; $\mathcal{P}$ is the normalized prompt embedding, produced by RepRTA from text or SAVPE from visual cues. The three prompt modes differ only in where $\mathcal{P}$ comes from.

Re-parameterizable region-text alignment (RepRTA) — text prompts

Text prompts are encoded with a frozen CLIP-family text encoder (MobileCLIP-B(LT)), $P=\text{TextEncoder}(T)\in {\mathbb{R}}^{C\times D}$. Instead of fusing text into image features, YOLOE leaves the image pathway alone and only refines the text embedding with a tiny auxiliary network $f_{\theta }$ — a single SwiGLU feed-forward block (two parallel Linear layers, elementwise product, Linear):

$\mathcal{P}=f_{\theta }(P)\in {\mathbb{R}}^{C\times D}$

• Training: all dataset text embeddings are cached in advance, so the text encoder is not even in the training loop; only $f_{\theta }$ is trained, adding negligible cost over closed-set training.
• Inference / transfer: $f_{\theta }(P)$ is a constant for a fixed vocabulary, so it is folded into the last conv of the embedding head. With $K\in {\mathbb{R}}^{D\times D^{\prime }\times 1\times 1}$ the conv kernel and $R$ a reshape,

$K^{\prime }=R_{C\times D\to C\times D\times 1\times 1}(f_{\theta }(P))⊛K^T,\text{Label}=I⊛K^{\prime }$

The result is an ordinary YOLO classification head with $C$ output channels — identical to a closed-set YOLO, zero overhead, and directly usable for downstream fine-tuning.

Figure 3 (paper): (a) RepRTA’s auxiliary network; (b) SAVPE’s semantic and activation branches.

Semantic-activated visual prompt encoder (SAVPE) — visual prompts

A visual prompt is a box or mask marking an example of the category. SAVPE turns it into a prompt embedding with two decoupled lightweight branches, avoiding deformable attention or a separate CLIP-style image encoder.

• Semantic branch — prompt-agnostic. Per scale, two $3\times 3$ convs on $P_3,P_4,P_5$; upsample, concatenate, project with $1\times 1$ to $S\in {\mathbb{R}}^{D\times H\times W}$. Same structure as the object embedding head.
• Activation branch — prompt-aware, in low dimension. The prompt is rasterized into a mask (1 inside the region, 0 elsewhere), downsampled and passed through a $3\times 3$ conv to give $F_V$; image features $F_I\in {\mathbb{R}}^{A\times H\times W}$ come from $1\times 1$ convs on $P_3,P_4,P_5$. They are concatenated and convolved into weights $\mathcal{W}\in {\mathbb{R}}^{A\times H\times W}$, softmax-normalized inside the prompt region.
• Grouped aggregation. The $D$ channels of $S$ are split into $A$ groups of $D/A$ channels; group $i$ shares the weight map ${\mathcal{W}}_{i:i+1}$:

$\mathcal{P}=\text{Concat}(G_1,\dots ,G_A),G_i={\mathcal{W}}_{i:i+1}\cdot S_{\frac{D}{A}\ast i:\frac{D}{A}\ast (i+1)}^T$

Because $A\ll D$, the visual-cue/image interaction happens in few channels, so it is cheap; the semantic branch is reused at full width. Default $A=16$. Compared with plain mask pooling of the semantic features, weighting positions inside the region matters (+1.5 AP, see ablations).

Lazy region-prompt contrast (LRPC) — prompt-free

Prompt-free detection must name everything in the image. Prior work generates names with a language model; YOLOE reformulates it as retrieval against a built-in vocabulary (a 4585-category tag list collected following [16] in the paper).

1. Train a specialized prompt embedding ${\mathcal{P}}_s$ that treats every object as one category (“object”), so it only finds objects.
2. At inference, keep only anchor points that are objects: ${\mathcal{O}}^{\prime }=\{o\in \mathcal{O}\mid o\cdot {\mathcal{P}}_s^T>\delta \}$.
3. Only those anchors are matched against the large vocabulary to retrieve a name.

“Lazy” = most anchor points are background, so the expensive $N\times C_{\text{vocab}}$ contrast is skipped for them. Same accuracy as contrasting everything against the vocabulary, but 1.7× (S) / 1.3× (L) faster.

Training

• Objective: as in YOLO-World, each mosaic sample gets an online vocabulary from the texts in its images (positive labels + sampled negatives). Labels are assigned with Task-Aligned-Assignment; BCE for classification, IoU + distributed focal loss for boxes, BCE for masks (as in YOLOv8-seg).
• Staged, each stage trains only what it needs:
  1. Text-prompt training of the whole model (30 epochs).
  2. Only SAVPE trained for 2 epochs with visual prompts; everything else frozen.
  3. Only the specialized prompt embedding trained for 1 epoch for prompt-free; everything else frozen.
• Data: detection/grounding — Objects365 (V1), GoldG (incl. GQA, Flickr30k), COCO images excluded. Segmentation masks are pseudo-labeled by SAM-2.1 from the GT boxes, then filtered/simplified. Visual-prompt data reuses GT boxes as cues; prompt-free data relabels all objects as a single class.

Hyperparameters

As reported in the paper (no separate optimizer table is given):

Item	Value
Architecture	YOLOv8 and YOLO11, scales S / M / L
Text encoder	MobileCLIP-B(LT), frozen, embeddings cached
SAVPE groups $A$	16
Prompt-free threshold $\delta$	0.001
Built-in vocabulary	4585 categories
Text-prompt training	30 epochs (vs 100 for YOLO-Worldv2)
Visual-prompt training (SAVPE only)	2 epochs
Prompt-free training (specialized embedding only)	1 epoch
Hardware	8× RTX 4090
Training time, text prompts (v8-S / M / L)	12.0 / 17.0 / 22.5 h
Training time, text prompts (11-S / M / L)	13.0 / 18.5 / 23.5 h
Visual-prompt eval	$N=16$ random training images per category, averaged embeddings
Negatives	global dictionary of category names appearing >100 times in training data

COCO transfer:

Strategy	Trainable	Epochs
Linear probing	classification head only	10 (all models)
Full tuning	everything	160 (v8-S / 11-S), 80 (M, L)

Results

Zero-shot LVIS detection (minival, Fixed AP, text prompt; FPS on T4 w/ TensorRT / iPhone 12 w/ CoreML):

Model	Train time	FPS (T4 / iPhone)	AP	AP${}_r$	AP${}_c$	AP${}_f$
YOLO-Worldv2-S	41.7 h	216.4 / 48.9	24.4	17.1	22.5	27.3
YOLO-Worldv2-M	60.0 h	117.9 / 34.2	32.4	28.4	29.6	35.5
YOLO-Worldv2-L	80.0 h	80.0 / 22.1	35.5	25.6	34.6	38.1
YOLOE-v8-S	12.0 h	305.8 / 64.3	27.9	22.3	27.8	29.0
YOLOE-v8-M	17.0 h	156.7 / 41.7	32.6	26.9	31.9	34.4
YOLOE-v8-L	22.5 h	102.5 / 35.9	35.9	33.2	34.8	37.3

• Gains are largest on rare categories (AP${}_r$ +5.2 for S, +7.6 for L vs YOLO-Worldv2); AP${}_f$ is slightly lower for M/L, which the authors attribute to multi-task (detection + segmentation) trade-off (see ablation).
• Visual prompts (T/V entries): YOLOE-v8-L beats T-Rex2 by +3.3 AP${}_r$ and +0.9 AP${}_c$ with about half the training data (1.4M vs 3.1M images) and far less compute; visual prompts help most on rare categories that text describes poorly.
• Segmentation (LVIS val, zero-shot AP${}^m$): YOLOE-v8-M / L reach 20.8 / 23.5 (text) — +3.0 / +3.7 over YOLO-Worldv2-M / L, which were fine-tuned on LVIS-Base for segmentation.
• Prompt-free (LVIS minival): YOLOE-v8-L 27.2 AP / 23.5 AP${}_r$ with 47M params at 25.3 FPS (PyTorch, T4), vs GenerateU (Swin-T) 26.8 / 20.0 with 297M params at 0.48 FPS — +0.4 AP, +3.5 AP${}_r$, 6.3× fewer parameters, ~53× faster.

COCO transfer (Table 4; AP${}^b$ / AP${}^m$):

Model	Setting	Epochs	AP${}^b$	AP${}^m$
YOLOv8-S	from scratch	500	44.7	36.6
YOLOE-v8-S	linear probing	10	35.6	30.3
YOLOE-v8-S	full tuning	160	45.0	36.7
YOLOv8-M	from scratch	300	50.0	40.3
YOLOE-v8-M	linear probing	10	42.2	35.5
YOLOE-v8-M	full tuning	80	50.4	40.9
YOLOv8-L	from scratch	300	52.4	42.3
YOLOE-v8-L	linear probing	10	45.4	38.3
YOLOE-v8-L	full tuning	80	53.0	42.7

With ~4× fewer epochs YOLOE-v8-M/L exceed YOLOv8-M/L by 0.4/0.6 AP${}^b$ and 0.6/0.4 AP${}^m$ (S: 3× less training time, also better). Linear probing (10 epochs, <2% of the training time) already reaches >80% of the closed-set model’s performance.

Ablations

Roadmap from YOLO-Worldv2-L to YOLOE-v8-L (LVIS minival AP, text prompts):

Step	AP	FPS T4 / iPhone
YOLO-Worldv2-L (100 ep)	33.0	80.0 / 22.1
+ fewer epochs (30)	31.0	80.0 / 22.1
+ global negative dictionary	31.9	80.0 / 22.1
− cross-modality fusion	30.0	102.5 / 27.2
+ MobileCLIP text encoder	31.5	102.5 / 27.2
+ RepRTA	33.5	102.5 / 27.2
+ segmentation head (= YOLOE)	33.3	102.5 / 27.2

Removing cross-modal fusion costs 1.9 AP but buys 1.28× / 1.23× speed; a stronger text encoder and RepRTA recover it (+2.3 AP from RepRTA alone, at zero inference cost). Adding the segmentation head costs only 0.2 AP (but 0.9 AP${}_f$).

SAVPE: activation branch vs mask pooling: 31.9 vs 30.4 AP. Group count: $A=1$ 30.9, $A=16$ 31.9, $A=32$ 31.9 AP — $A=16$ is the balance point.

LRPC (v8-S): no LRPC 21.0 AP at 56.5 FPS; $\delta =10^{-3}$ 21.0 AP at 95.8 FPS; $\delta =10^{-2}$ 20.8 AP at 106 FPS (1.9× speedup for 0.2 AP). For v8-L: 27.2 AP at 19.9 → 25.3 FPS.

Limitations

• Detection/segmentation multi-task trade-off: joint training lowers frequent-category AP${}_f$ slightly (YOLOE-v8-M/L trail YOLO-Worldv2-M/L on AP${}_f$).
• Segmentation masks are SAM-2.1 pseudo-labels from boxes, so mask quality is bounded by that teacher.
• Prompt-free names come from a fixed 4585-word vocabulary; anything outside it cannot be named.
• Visual prompts need example images per category (16 by default) to build a stable embedding.

See also

• YOLOv8 — the architecture YOLOE is built on.
• YOLOv10 — same lab, NMS-free closed-set YOLO.
• Task-Aligned-Assignment — label assignment used in training.
• YOLACT — the prototype + coefficient mask head reused for segmentation.
• SAM3 — concept-promptable segmentation (text or exemplar) at a very different scale and cost.
• DE-ViT — few-shot detection from frozen DINOv2 features, the visual-prompt counterpart without a YOLO.
• Segmentation Head Comparison — how the mask head fits among other designs.