Co-Training
Co-Training is a semi-supervised learning algorithm that trains two models on two different "views" (independent feature sets) of the same data, with each model teaching the other by labeling its most confident predictions — exploiting the principle that when two sufficient and independent views agree on an unlabeled example, that prediction is highly reliable, enabling learning from very small labeled datasets by leveraging the structure of multi-view data.
What Is Co-Training?
- Definition: A semi-supervised method (Blum & Mitchell, 1998) that splits features into two independent subsets (views), trains a separate classifier on each view, and iteratively expands the labeled set by having each classifier label the examples it is most confident about for the other classifier.
- The Key Insight: If two different feature sets independently support the same prediction, that prediction is almost certainly correct. This "agreement" signal from independent views is stronger than any single model's confidence.
- The Requirement: The two views must be (1) sufficient — each view alone can learn a good classifier, and (2) conditionally independent — given the label, the views provide independent evidence.
The Classic Example: Web Page Classification
| View | Features | Rationale |
|---|---|---|
| View 1 (Content) | Text on the web page itself | Describes the page's own content |
| View 2 (Links) | Anchor text of hyperlinks pointing TO the page | Describes how others perceive the page |
These views are naturally independent — what a page says about itself vs. what other pages say about it.
Co-Training Algorithm
| Step | Action | Result |
|---|---|---|
| 1. Initialize | Train Model A on View 1 (labeled data), Model B on View 2 (labeled data) | Two weak classifiers |
| 2. Predict | Each model predicts labels for all unlabeled examples | Confidence scores for each example |
| 3. Select | Each model picks its top-k most confident predictions | High-confidence pseudo-labels |
| 4. Teach | Add Model A's confident examples to Model B's training set (and vice versa) | Expanded training sets |
| 5. Retrain | Retrain both models on their expanded training sets | Improved classifiers |
| 6. Repeat | Iterate steps 2-5 until convergence or budget exhausted | Progressively better models |
Why Two Models Beat One
| Scenario | Single Model (Self-Training) | Co-Training (Two Views) |
|---|---|---|
| Error propagation | Model reinforces its own mistakes | Independent views catch each other's errors |
| Diversity | One perspective on the data | Two complementary perspectives |
| Confirmation bias | High risk — same model generates and learns from pseudo-labels | Lower risk — different feature spaces reduce correlated errors |
| Requirement | Any features | Needs two sufficient, independent views |
Co-Training vs Other Semi-Supervised Methods
| Method | Approach | Key Advantage | Limitation |
|---|---|---|---|
| Co-Training | Two models on two views teach each other | Exploits multi-view structure, reduces confirmation bias | Requires naturally independent feature views |
| Self-Training | One model labels its own data | Simplest approach, no view requirement | High confirmation bias risk |
| Pseudo-Labeling | Hard labels from confident predictions | Framework-agnostic | Same bias as self-training |
| MixMatch | Consistency regularization + pseudo-labels | State-of-the-art accuracy | Complex implementation |
| Label Propagation | Graph-based label spreading | Works with any similarity metric | Expensive for large datasets |
Real-World Applications
| Domain | View 1 | View 2 |
|---|---|---|
| Web classification | Page text content | Inbound link anchor text |
| Email spam | Email body text | Email header metadata |
| Named entity recognition | Local word context | Broader document context |
| Image + text | Image features | Caption text |
| Medical imaging | MRI scan | Patient clinical notes |
Co-Training is the foundational multi-view semi-supervised learning algorithm — leveraging the agreement between two independent feature views to generate reliable pseudo-labels with lower confirmation bias than single-model self-training, enabling effective learning from tiny labeled datasets when data naturally admits two sufficient and independent views.