Image Cutout › Methodology
How Image Cutout works, and how its accuracy was verified
Last verified 2026-09-04 · corrections are logged in the change log.
Summary. Image Cutout runs Meta's Segment Anything in the browser. You click a point, the model returns a mask for the object under it, and the mask becomes the alpha channel of a PNG. Nothing is uploaded. The sections below give the exact pipeline, the model versions, the measurements behind the accuracy and speed claims made elsewhere on this site, and the limits of the method.
The pipeline, step by step
- Preprocess (once per photo). The image is scaled so its longest side is 1024 px, normalised with the standard ImageNet mean and standard deviation, and zero-padded to a 1024x1024 square at the bottom and right.
- Encode (once per photo). The vision encoder turns that tensor into an image embedding and a positional embedding, each 1x256x64x64. This is the expensive step, and its result is cached for as long as the photo stays open.
- Decode (once per click). Your points are converted into the same 1024-px space and passed, with both embeddings, to the prompt encoder and mask decoder. It returns three candidate masks as 256x256 logits plus a predicted quality score for each.
- Choose and resample. The candidate with the highest predicted score is upsampled to 1024x1024, cropped back to the unpadded region, resized to the photo's real dimensions, and thresholded at zero with a one-logit soft edge.
- Composite. The result becomes the alpha channel of the original full-resolution pixels, optionally over a chosen background, and is encoded as PNG or JPG by the browser.
Models and provenance
Both models are third-party open-source ONNX exports, pinned to an exact commit and served from SkillSafe's shared registry. The browser verifies the SHA-256 of every downloaded file against a same-origin manifest before handing the bytes to the runtime, so a substituted or corrupted weight file cannot be used.
| Model | Role | Source | Pinned commit | Licence |
|---|---|---|---|---|
| SlimSAM-77 (uniform) | Default. A pruned SAM whose encoder is in the ViT-Tiny size class, about 9M parameters | Xenova/slimsam-77-uniform | 5850ab45 | Apache-2.0 |
| SAM ViT-Base | Optional. Meta's original ViT-B image encoder, about 86M parameters | Xenova/sam-vit-base | e9ff25bc | Apache-2.0 |
Both share the same prompt encoder and mask decoder; SlimSAM prunes only the image encoder, so those two files are byte-identical and are downloaded once. Exact file names, byte counts and SHA-256 digests are published at /models/NOTICE.txt. The runtime is onnxruntime-web 1.27.0 (MIT), vendored unmodified.
How the accuracy was checked
The browser is not trusted to be correct by assumption. A second, independent implementation of the same pipeline was written in Python against onnxruntime on the CPU, using the reference image-processor maths rather than the browser code, and the two masks were compared by intersection over union. Agreement measured on 2026-09-04:
| Model and precision | Runtime | Photo and prompt | IoU vs reference |
|---|---|---|---|
| SlimSAM-77 fp16 | WebGPU | Portrait, 1 point | 0.9993 |
| SlimSAM-77 int8 | WebAssembly | Portrait, 1 point | 0.9983 |
| SlimSAM-77 fp16 | WebGPU | Two cups, 2 positive + 1 negative | 0.9876 |
| SAM ViT-Base fp16 | WebGPU | Portrait, 1 point | 0.9983 |
| SAM ViT-Base int8 | WebAssembly | Two cups, 2 positive + 1 negative | 0.9936 |
| SAM ViT-Base fp16 | WebGPU | Two cups, 2 positive + 1 negative | 0.9274 |
The last row is reported deliberately rather than omitted. Where the correct boundary is genuinely ambiguous — in that photo, a stream of coffee touching the cup — the two implementations resolve the ambiguity differently, and the disagreement is confined to that region. On unambiguous selections the two agree to about a thousandth.
A separate check confirms that zooming changes nothing: the identical image point produces a byte-identical mask at best fit and at 3x zoom.
Measured speed
Times for a 1400x1096 photo, in Chrome. The encode runs once per photo; each later click only re-runs the decoder.
| Model | Runtime | Encode | Per click |
|---|---|---|---|
| SlimSAM-77 fp16 | WebGPU | 0.5-0.9 s | 35-60 ms |
| SAM ViT-Base fp16 | WebGPU | 1.0-1.1 s | about 35 ms |
| SlimSAM-77 int8 | WebAssembly, single thread | 3.5-3.9 s | 110-140 ms |
| SAM ViT-Base int8 | WebAssembly, single thread | about 24 s | about 130 ms |
Hardware and browser change these figures. WebAssembly is single-threaded here because the page is not cross-origin isolated.
Known limits
- Edges are Segment-Anything quality, not matting: clean on solid objects, soft on hair and fur. There is no alpha-matting refinement stage.
- The model works at 1024 px on the long side and the mask is resampled up, so very fine structures in a large photo are limited by that working resolution.
- Photos above 4096 px on the long side are reduced to 4096 px before anything else.
- Quantised (int8) weights trade a little accuracy for size and are what runs when WebGPU is unavailable.
- SAM ViT-Base is slow without WebGPU, which is why the smaller model is the default.
Corrections
If a figure here is wrong, it will be corrected and the correction dated in the change log rather than edited away silently. Reports: gptsci.media@gmail.com.
FAQ
How accurate is the in-browser segmentation?
Browser masks match an independent Python implementation of the same pipeline to an IoU of about 0.99 or better on unambiguous selections. Where the true boundary is ambiguous the two can differ more; a measured example is published on the methodology page.
Which models does Image Cutout use?
SlimSAM-77 uniform by default and, optionally, SAM ViT-Base. Both are Apache-2.0 ONNX exports pinned to an exact commit, and the browser verifies each file's SHA-256 before use.
How fast is it?
With WebGPU, encoding a 1400x1096 photo takes about 0.5 to 0.9 seconds with SlimSAM-77 and about 1.1 seconds with SAM ViT-Base; each click after that takes tens of milliseconds.