Meta-Dataset
Meta-Dataset is a large-scale benchmark for evaluating few-shot learning algorithms, consisting of a diverse collection of datasets spanning different visual domains. Introduced by Triantafillou et al. (2020), it addressed critical limitations of earlier single-domain evaluations.
Why Meta-Dataset Was Needed
- Single-Domain Limitation: Earlier benchmarks (miniImageNet, Omniglot) evaluated few-shot learning within a single visual domain. Models could achieve high accuracy by learning domain-specific features rather than general few-shot learning strategies.
- Fixed Episode Structure: Standard benchmarks used fixed 5-way 5-shot or 5-way 1-shot episodes, which doesn't reflect real-world variability.
- Overfit to Benchmark: Many methods were optimized specifically for miniImageNet, achieving high scores without truly general few-shot capabilities.
Component Datasets (10 Domains)
| Domain | Dataset | Classes | Description |
|---|---|---|---|
| Natural Images | ImageNet | 1,000 | General object recognition |
| Handwriting | Omniglot | 1,623 | Handwritten characters from 50 alphabets |
| Aircraft | FGVC-Aircraft | 100 | Fine-grained aircraft model recognition |
| Birds | CUB-200 | 200 | Fine-grained bird species |
| Textures | DTD | 47 | Describable texture patterns |
| Drawings | Quick Draw | 345 | Hand-drawn sketches |
| Fungi | FGVCx Fungi | 1,394 | Mushroom species identification |
| Flowers | VGG Flower | 102 | Flower species recognition |
| Signs | Traffic Signs | 43 | Traffic sign classification |
| Objects | MSCOCO | 80 | Object categories in context |
Key Design Innovations
- Variable-Way Variable-Shot: Episodes have variable numbers of classes and examples per class — reflecting realistic scenarios where you might have 3 examples of one class and 10 of another.
- Realistic Distributions: Class and sample counts follow realistic distributions rather than fixed configurations.
- Cross-Domain Evaluation: Train on a subset of datasets, test on held-out datasets to measure generalization to entirely new visual domains.
- Within-Domain Testing: Also evaluate on unseen classes from training datasets to measure both cross-domain and within-domain generalization.
Evaluation Protocol
- Training Sources: Typically train on ImageNet, Omniglot, Aircraft, CUB-200, DTD, Quick Draw, Fungi, VGG Flower.
- Test Sources: Evaluate on held-out test classes from training datasets PLUS entirely unseen datasets (Traffic Signs, MSCOCO).
- Metric: Average accuracy across many sampled episodes, reported per dataset.
Key Findings
- Many methods optimized for miniImageNet performed poorly across diverse domains — exposing the limitation of single-domain benchmarks.
- Large pre-trained feature extractors significantly outperformed meta-learning methods trained from scratch.
- Universal representations (features that work across all domains) are more effective than domain-specific adaptation for most target domains.
Meta-Dataset established the gold standard for few-shot learning evaluation — any new few-shot method must demonstrate effectiveness across its diverse domains to be considered truly general.