Falcon

Falcon LLM (TII)

Overview
Falcon is a family of open-source Large Language Models developed by the Technology Innovation Institute (TII) in Abu Dhabi. Upon release (May 2023), Falcon-40B was the top-ranked open-source model on the Hugging Face Open LLM Leaderboard.

Key Models
- Falcon-180B: A massive model comparable to GPT-3.5/GPT-4 in some benchmarks. Released Sept 2023.
- Falcon-40B: The original flagship.
- Falcon-7B: Lightweight version.

Architecture
Falcon introduced Multi-Query Attention (MQA) to the mainstream.
- Standard Attention: One head per query/key/value.
- MQA: Shared keys/values across all heads.
- Result: Significantly faster inference and lower memory usage (KV Cache) during text generation.

Data Quality: RefinedWeb
TII claimed the secret sauce was their RefinedWeb dataset. They filtered CommonCrawl aggressively, proving (like Dolly) that clean data beats "more data."

Licensing
Originally restrictive, TII re-licensed Falcon under Apache 2.0 (fully open metric) due to community feedback, setting a precedent for Open Weights.