Mpt

MPT: Mosaic Pretrained Transformer

Overview
MPT is a series of open-source LLMs created by MosaicML (acquired by Databricks). They were designed to showcase Mosaic's efficient training infrastructure.

Key Innovations

1. ALiBi (Attention with Linear Biases)
MPT does not use standard Positional Embeddings. It uses ALiBi.
- Benefit: The model can extrapolate to context lengths *longer* than it was trained on.
- MPT-7B-StoryWriter could handle 65k context length (massive for early 2023) on consumer GPUs.

2. Training Efficiency
MPT was trained from scratch in roughly 9 days for $200k. It demonstrated that training "foundational models" was within reach of startups, not just Google/OpenAI.

3. Commercial License
MPT-7B released with an Apache 2.0 license immediately, allowing commercial use (unlike LLaMA 1 which was research only).

Models
- MPT-7B: Base model.
- MPT-30B: Higher quality, rivals GPT-3.

Legacy
MPT pushed the industry toward longer context windows and faster attention mechanisms (FlashAttention integration).