Open Image Segmentation Datasets: Semantic, Instance, Panoptic
COCO covers all three segmentation tasks on one set of images and is the standard starting point; add ADE20K for scene-level semantic diversity, Cityscapes for driving scenes, LVIS for long-tailed rare categories, and SA-1B for very large-scale class-agnostic masks. Below is what each dataset actually contains, its license, and how to pick between them by task and domain.
The short answer
Pick by task first, then by domain. For a general-purpose model covering semantic, instance and panoptic segmentation together, use COCO. For indoor and outdoor scene diversity beyond everyday objects, use ADE20K. For urban driving scenes, use Cityscapes or Mapillary Vistas. For long-tailed, rare-category instance segmentation, use LVIS. For class-agnostic segmentation at very large scale, use SA-1B. Check both the image license and the annotation license before commercial use — they are often different.
General-purpose segmentation
COCO (Common Objects in Context)
COCO is the reference dataset for segmentation research: 118k training images and 5k validation images with instance segmentation across 80 "thing" categories, plus panoptic annotations that add 53 "stuff" categories (COCO-Panoptic) for a combined 133-category panoptic task. Nearly every pretrained instance and panoptic segmentation model reports a COCO benchmark, so it is the easiest dataset to compare against existing work. COCO annotations are released under a Creative Commons license, but the underlying images remain subject to Flickr's terms of use since COCO does not redistribute image copyright. Get it from cocodataset.org.
COCO-Stuff
COCO-Stuff extends the same 164k COCO images with dense pixel-level "stuff" annotations — background material classes like sky, grass, water and wall — for a combined 172 classes (80 thing, 91 stuff, 1 unlabeled). It is the dataset to use when your task needs full-scene semantic coverage, not just the countable foreground objects COCO's original instance labels focus on. The annotations and code are Creative Commons Attribution 4.0; images still follow Flickr's terms via COCO. Get it from github.com/nightrome/cocostuff.
Scene-diverse semantic segmentation
ADE20K
ADE20K, released by MIT, covers over 20,000 training images and 2,000 validation images across 150 semantic categories, spanning indoor scenes like kitchens and bedrooms to outdoor scenes like mountains and beaches. Its strength is scene diversity rather than object-category breadth, which makes it useful when a model needs to generalize beyond COCO's everyday-object focus into full scene parsing. Get it from the official MIT site at ade20k.csail.mit.edu, and check the license terms on the download page before commercial use.
Driving and urban scenes
Cityscapes
Cityscapes is built specifically for urban driving scene understanding: 5,000 images with fine pixel-level annotation (2,975 train / 500 val / 1,525 test) across 30 semantic classes, plus 20,000 additional images with coarse annotation, captured across 50 different cities with stereo pairs and GPS metadata. It supports pixel-level, instance-level and panoptic labeling benchmarks in the same release, which makes it the standard reference for autonomous-driving perception research. It is licensed for non-commercial academic and research use only. Get it from cityscapes-dataset.com (registration required).
Mapillary Vistas
Mapillary Vistas is a street-level imagery dataset for semantic, instance and panoptic segmentation, built from crowd-sourced photos captured across a much wider range of countries, weather and lighting conditions than Cityscapes' fixed set of cities, which makes it useful for testing how well a driving-scene model generalizes geographically. Review the current image count, class count and license terms on the official page before use, since Mapillary periodically updates the release. Get it from mapillary.com/dataset/vistas.
Long-tail and very large-scale segmentation
LVIS
LVIS ("Large Vocabulary Instance Segmentation") reuses COCO's images but adds instance segmentation masks for over 1,000 object categories with a long-tailed frequency distribution, meaning most categories have only a handful of labeled examples. It is the dataset to use when you need to evaluate or train for rare-category detection and segmentation rather than the small, well-balanced category set COCO provides. Get it from lvisdataset.org.
SA-1B (Segment Anything)
SA-1B is Meta's dataset behind the Segment Anything Model, containing 11 million high-resolution images and 1.1 billion segmentation masks — roughly 100 masks per image — generated automatically and reviewed at scale. Masks are class-agnostic, meaning they mark object boundaries without assigning a category label, which makes SA-1B suited to pretraining a general-purpose segmentation backbone rather than a fixed-category classifier. Meta licenses it for research purposes only. Get it from ai.meta.com/datasets/segment-anything.
Datasets side by side
| Dataset | Segmentation type | Scale | License |
|---|---|---|---|
| COCO | Instance + panoptic | 118k train / 5k val, 80–133 classes | CC (annotations); Flickr ToU (images) |
| COCO-Stuff | Semantic (stuff) | 164k images, 172 classes | CC BY 4.0 (annotations) |
| ADE20K | Semantic, scene-diverse | 20k train / 2k val, 150 classes | Check download page |
| Cityscapes | Semantic + instance + panoptic | 5k fine / 20k coarse, 30 classes | Non-commercial research only |
| Mapillary Vistas | Semantic + instance + panoptic | Check official page | Check official page |
| LVIS | Instance, long-tail | 1,000+ categories on COCO images | Check official page |
| SA-1B | Class-agnostic mask | 11M images, 1.1B masks | Research use only |
How to choose
If you are building or evaluating a general-purpose segmentation model, start with COCO because it is the most widely benchmarked and gives you all three task types on one image set. If your application is a driving or robotics perception system, Cityscapes and Mapillary Vistas better match the domain than COCO's everyday-object photos. If your model needs to recognize rare or long-tailed categories, LVIS reveals failure modes COCO's balanced categories hide. If you are pretraining a segmentation backbone rather than a fixed classifier, SA-1B's scale and class-agnostic masks are more useful than any labeled category set. Whatever you choose, budget time to reconcile license terms between the images and the annotations — they are frequently governed by different licenses within the same dataset.
FAQ
What is the difference between semantic, instance and panoptic segmentation datasets?
Semantic segmentation labels every pixel with a class without separating individual objects of the same class. Instance segmentation labels and separates each individual object, typically only for countable "thing" categories. Panoptic segmentation combines both: every pixel gets a class label, and countable objects are separated into distinct instances. COCO-Panoptic and Cityscapes support panoptic evaluation directly; ADE20K and COCO-Stuff are primarily semantic; COCO and LVIS are primarily instance-focused.
Which segmentation dataset should I start with for a general-purpose model?
COCO is the most widely used starting point because it has instance, panoptic and stuff annotations on the same 118k training images, broad tooling support, and most pretrained segmentation models report benchmarks on it. If your model needs to generalize to scenes far outside COCO's everyday-object categories, add ADE20K or SA-1B rather than replacing COCO.
Can I use these segmentation datasets commercially?
It varies by dataset and even by asset type within a dataset. COCO annotations and COCO-Stuff annotations are CC BY 4.0, but the underlying COCO images remain under Flickr's terms of use. Cityscapes is restricted to non-commercial academic and research use. SA-1B images are commercially licensed from a photo provider but Meta restricts the dataset's intended use to research. Always check the specific license page for both images and annotations before commercial use.
Still cannot find the right segmentation data? Leave it to us
If your project needs segmentation data for a domain none of these cover — medical imaging, satellite imagery, industrial inspection — tell us your task goal and the conditions it must meet. DeepSData can run a real availability search and verify sources for you, checking reputable data platforms and judging matches and gaps against each requirement, with every source link kept traceable.
New accounts get 3 free searches; every result comes with verifiable source links. See also curated datasets.