MobileSAM
MobileSAM is a lightweight re-training of SAM for resource-constrained devices.
- Paper: Zhang, Han, Qiao, Kim, Bae, Lee, Hong (Kyung Hee University), Faster Segment Anything: Towards Lightweight SAM for Mobile Applications, July 2023, arXiv:2306.14289.
- Architecture: keeps SAM’s pipeline exactly (ViT image encoder → prompt encoder → mask decoder) and swaps only the image encoder for a 5M-parameter ViT-Tiny.
- Training trick: decoupled distillation from SAM’s ViT-H encoder, on a single GPU in under a day.
- Headline result: ~60× smaller in total parameters and ~10 ms per image, while performing on par with the original SAM.
Motivation: the encoder is the whole problem
SAM’s parameter count is almost entirely its encoder. From Table 1 of the paper:
| Component | SAM (ViT-H) | SAM (ViT-L) | SAM (ViT-B) |
|---|---|---|---|
| ViT-based image encoder | 632M | 307M | 86M |
| Prompt-guided encoder | 0.006M | 0.006M | 0.006M |
The mask decoder is 3.87M parameters and the prompt encoder 0.006M, so given a precomputed image embedding SAM already runs comfortably on a phone — this is what the official web demo exploits. The heavyweight part is the 632M-parameter ViT-H encoder that must run once per image. Making SAM mobile-friendly therefore reduces entirely to replacing that encoder with a small one, which automatically preserves every other capability of the model.
The obvious route — retrain SAM end-to-end with a smaller backbone, as in the original paper — is out of reach: SAM with ViT-H took 68 hours on 256 A100s, and even the ViT-L/ViT-B variants need 128 GPUs for multiple days. Worse, the authors find that naively retraining this way gives unsatisfactory performance when training data is limited, and diagnose the cause as the coupled optimization of image encoder and mask decoder.
Coupled vs. decoupled distillation
Following the original SAM recipe with a small encoder is, in essence, knowledge distillation from a ViT-H-based teacher SAM to a student SAM — the “ground-truth” masks in SA-1B were themselves generated by the pretrained SAM. Three variants exist:
- Fully-coupled distillation. Student encoder and student decoder are trainable, and the distillation signal is applied at the predicted mask. Encoder quality depends on decoder quality and vice versa; when both modules start in a bad state, driving them both to a good one is hard.
- Semi-coupled distillation. Copy the teacher’s decoder into the student and freeze it, optimizing only the encoder against the mask output. Freezing prevents a poor encoder from degrading the decoder, but the optimization is still not decoupled: the choice of prompt is random, which makes the effective decoder behaviour variable and keeps optimization difficult.
- Decoupled distillation (proposed). Distil the student encoder directly against the teacher’s image embedding, with no decoder in the loop at all.
Decoupling splits the KD task into two sub-tasks — image-encoder distillation and (optional) mask-decoder finetuning — in the spirit of divide-and-conquer. Two practical benefits follow:
- The objective becomes a plain MSE loss on the image embeddings, instead of the combination of focal and dice losses needed for mask prediction in SAM.
- Prompts never enter training, so nothing is random about the target.
On the necessity of decoder finetuning. A decoupled student encoder might in principle misalign with the frozen original mask decoder. Empirically it does not: the student’s image encoding lands close enough to the teacher’s that the second-stage finetuning on the combined decoder is optional, and the released MobileSAM omits it. Because the lightweight encoder is distilled from the very encoder the original decoder was trained against, alignment comes for free — this is also what makes MobileSAM plug-and-play for existing SAM-based projects, which need only swap the encoder.
Preliminary evaluation
Compared under SAM with ViT-B as the student encoder, measuring mIoU between teacher and student masks on the same prompt point (averaged over 200 samples):
| Coupled distillation | Decoupled distillation | |
|---|---|---|
| mIoU | 0.72 | 0.75 |
| Training GPUs | 128 | 2 |
| Batch size | 128 | 4 |
| Iterations | 180k | 55k |
| Training data | 11M images | 11k images (0.1% of SA-1B) |
Decoupled distillation is better and uses less than 1% of the compute. Since ViT-B is still too heavy for mobile, the final MobileSAM uses a 5M-parameter ViT-Tiny instead.
The lightweight image encoder
MobileSAM adopts ViT-Tiny from TinyViT (Wu et al., ECCV 2022), which has parameter counts similar to Deit-Tiny but performs much better (79.1% vs. 72.2% ImageNet-1K top-1). Its structure:
- Four stages that progressively reduce resolution.
- The first stage is built from convolution blocks with inverted residuals (MobileNetV2-style); the remaining three stages are transformer blocks.
- Two convolution blocks with stride 2 at the very beginning downsample the input.
- Downsampling between stages is done by stride-2 convolution blocks — except that, unlike TinyViT, the stride of the last downsampling convolution is set to 1, so the final feature resolution matches that of SAM’s ViT-H encoder (and therefore the frozen decoder’s expectation).
Any other efficient ViT (EfficientFormer, EfficientViT, Next-ViT, MobileViT) could be dropped in the same way; the distillation recipe is orthogonal to the backbone choice.
| Original SAM | MobileSAM | |
|---|---|---|
| Encoder parameters | 632M | 5.78M |
| Encoder speed (single GPU) | 452 ms | 8 ms |
Total inference is ~10 ms per image: 8 ms encoder + 4 ms decoder (the paper quotes both the ~10 ms total and the two components; the decoder figure is measured separately). Total parameters drop ~60×, encoder parameters ~100×.
Hyperparameters
Transcribed from §4.1 of the paper:
| Student encoder | ViT-Tiny (TinyViT), 5.78M parameters |
| Teacher | image encoder of SAM (ViT-H) |
| Distillation target | teacher image embedding |
| Loss | simple MSE |
| Training data | 1% of SA-1B |
| Epochs | 8 |
| Hardware | a single GPU, < 1 day |
| Decoder | copied from original SAM, frozen; finetuning optional and omitted |
| Optimization trick | teacher image embeddings precomputed once and cached, since the forward pass through ViT-H dominates cost |
Ablation on training computation
| Batch size | Epochs | Iterations | mIoU |
|---|---|---|---|
| 4 | 2 | 50k | 0.7057 |
| 8 | 4 | 50k | 0.7286 |
| 8 | 8 | 100k | 0.7447 |
Both larger batches (at fixed iterations) and more iterations help; all runs were done on a single GPU, and the authors expect more GPUs — bigger batches, more iterations — to improve results further.
Results
Evaluation is by mIoU against the original SAM’s predicted mask treated as ground truth, i.e. how faithfully MobileSAM reproduces SAM, plus qualitative comparison. With point prompts (Fig. 4) and box prompts (Fig. 5), MobileSAM’s masks are close to the original SAM’s.
Versus FastSAM
FastSAM is the concurrent lightweight alternative, built very differently: a YOLOv8-based detection branch plus a YOLACT-based segmentation branch that generate mask proposals prompt-free, with promptable behaviour retrieved afterwards by a mapping algorithm that selects from the proposal set.
| FastSAM | MobileSAM | Ratio | |
|---|---|---|---|
| Size | 68M | 9.66M | ≈ 7× smaller |
| Speed | 64 ms | 12 ms | ≈ 5× faster |
mIoU against the original SAM’s masks, with two prompt points (one foreground, one background) at varying pixel distance:
| Point distance | 100 | 200 | 300 | 400 | 500 |
|---|---|---|---|---|---|
| FastSAM | 0.27 | 0.33 | 0.37 | 0.41 | 0.41 |
| MobileSAM | 0.73 | 0.71 | 0.74 | 0.73 | 0.73 |
FastSAM’s masks diverge sharply from SAM’s, and its mIoU degrades as the two prompt points get closer — it often fails to predict the object when the foreground point sits near the background point. MobileSAM is essentially flat in this parameter.
Segment anything vs. segment everything
The paper is careful about the distinction, which is also the basis of its FastSAM critique:
| anything | everything | |
|---|---|---|
| # of objects | 1 | N |
| prompt-aware | yes | no |
“Segment anything” is the foundation task — promptable segmentation returning a valid mask for any prompt. “Segment everything” is object proposal generation, a downstream task where no prompt is required. FastSAM targets the latter and reconstructs the former; MobileSAM inherits the former directly, which is why it stays compatible with the whole ecosystem of SAM-based work.
For completeness the paper also compares on segment-everything: MobileSAM’s proposals align surprisingly well with the original SAM’s, while FastSAM often misses objects entirely (the roof in the first image), produces hard-to-interpret proposals (the stage, the sky) and generates masks with non-smooth boundaries — visible on the pillars in the third row, where SAM and MobileSAM do not have the problem.
Limitations
- The evaluation uses the original SAM’s own predictions as ground truth, so it measures fidelity to SAM rather than absolute segmentation quality; no benchmark against human-annotated masks is reported.
- No text-prompt results, because the official SAM repository ships no pretrained text-guided decoder.
- Mask-decoder finetuning after distillation is expected to improve performance but is omitted from this version of the paper.
- Training was constrained to a single GPU; the ablation shows the model is not saturated at the reported compute budget.
- MobileSAM is an efficiency result only — it inherits every accuracy limitation of SAM (fine structures, boundary crispness, no semantics) and cannot exceed its teacher.
See also
- SAM — the teacher model and the pipeline MobileSAM preserves wholesale
- SAM2 — the video successor, which addresses a different axis (temporality, not cost)
- YOLOv8 — the detector underneath FastSAM, MobileSAM’s main comparison point
- GPUs for Local ML — what “runs on a single GPU in under a day” costs in practice
- Implementing Research Papers