Pretraining

Pre-training is the initial training phase where models learn general patterns from large unlabeled datasets — creating foundation models that capture broad language or vision understanding, which can then be fine-tuned for specific downstream tasks with much less data and compute.

What Is Pre-Training?

• Definition: Training on large, general datasets before specialization.
• Objective: Learn universal representations (language patterns, visual features).
• Scale: Billions of tokens/images, weeks-months of compute.
• Output: Foundation model or base model.

Why Pre-Training Works

• Transfer Learning: General knowledge transfers to specific tasks.
• Data Efficiency: Fine-tuning needs much less task-specific data.
• Emergence: Capabilities arise from scale that can't be directly trained.
• Cost Amortization: One expensive pre-train, many cheap fine-tunes.
• Better Representations: Self-supervised learning captures structure.

Pre-Training Objectives

Language Models:

Objective             | Description
----------------------|----------------------------------
Causal LM (GPT)       | Predict next token: P(x_t | x_{<t})
Masked LM (BERT)      | Predict masked tokens: P(x_mask | context)
Span corruption (T5)  | Predict multiple masked spans
Prefix LM             | Bidirectional attention on prefix

Vision Models:

Objective             | Description
----------------------|----------------------------------
Contrastive (CLIP)    | Match images to text descriptions
MAE                   | Reconstruct masked image patches
SimCLR                | Match augmented views of same image
DINO                  | Self-distillation without labels

Pre-Training at Scale

Modern LLM Training:

Component         | Typical Scale
------------------|---------------------------
Data              | 1-15 trillion tokens
Parameters        | 7B-405B parameters
Compute           | 10^23-10^25 FLOPs
Hardware          | 1000s-10000s of GPUs
Time              | Weeks to months
Cost              | $1M-$100M+

Dataset Composition:

Source              | Percentage
--------------------|------------
Web crawl (CommonCrawl)| 60-80%
Code (GitHub)       | 5-15%
Books               | 5-10%
Wikipedia           | 2-5%
Scientific papers   | 2-5%
Curated/synthetic   | Variable

Pre-Training Pipeline

Stages:

Code Example:

from transformers import (
    AutoModelForCausalLM, 
    AutoTokenizer,
    Trainer, 
    TrainingArguments,
    DataCollatorForLanguageModeling,
)

# Load base architecture
model = AutoModelForCausalLM.from_config(config)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Pre-training data
def tokenize(examples):
    return tokenizer(examples["text"], truncation=True, max_length=2048)

tokenized_dataset = dataset.map(tokenize, batched=True)

# Training arguments for pre-training
training_args = TrainingArguments(
    output_dir="./pretrained-model",
    per_device_train_batch_size=8,
    gradient_accumulation_steps=16,
    learning_rate=3e-4,
    warmup_steps=2000,
    max_steps=500000,
    bf16=True,
    save_steps=5000,
)

# Data collator for causal LM
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    data_collator=data_collator,
)

trainer.train()

Pre-Training vs. Fine-Tuning

Aspect          | Pre-Training      | Fine-Tuning
----------------|-------------------|------------------
Data            | Billions tokens   | Thousands-millions
Compute         | $1M+              | $10-$10K
Time            | Weeks-months      | Hours-days
Objective       | General LM        | Task-specific
Who does it     | AI labs           | Everyone
Learning rate   | Higher (1e-4)     | Lower (1e-5)

Pre-training is the foundation of modern AI — by investing massive resources once to create powerful general-purpose models, the community enables efficient specialization through fine-tuning, democratizing access to capabilities that would be impossible to train from scratch.