Domain Adaptation

Domain Adaptation

What is Domain Adaptation?
Techniques to transfer knowledge when source and target domains have different distributions, addressing the "domain shift" problem.

Types of Domain Shift
| Shift Type | Example |
|------------|---------|
| Covariate | Different input distributions |
| Label | Different class distributions |
| Concept | Same input, different meaning |
| Prior | Different class frequencies |

Domain Adaptation Scenarios
| Scenario | Source Labels | Target Labels |
|----------|---------------|---------------|
| Supervised | Yes | Yes |
| Semi-supervised | Yes | Few |
| Unsupervised | Yes | No |

Techniques

Feature Alignment
Learn domain-invariant features:

class DomainAdapter(nn.Module):
    def __init__(self, encoder, classifier, discriminator):
        self.encoder = encoder
        self.classifier = classifier
        self.discriminator = discriminator

    def forward(self, source, target):
        source_features = self.encoder(source)
        target_features = self.encoder(target)

        # Classification loss on source
        class_loss = criterion(self.classifier(source_features), labels)

        # Domain confusion loss (adversarial)
        domain_loss = domain_criterion(
            self.discriminator(source_features),
            self.discriminator(target_features)
        )

        return class_loss - lambda_ * domain_loss

Pseudo-Labeling
Use model predictions on target domain:

# Generate pseudo-labels
with torch.no_grad():
    target_preds = model(target_data)
    confidence, pseudo_labels = target_preds.max(dim=1)

    # Keep high-confidence predictions
    mask = confidence > threshold

# Train on pseudo-labeled targets
loss = criterion(model(target_data[mask]), pseudo_labels[mask])

Domain Randomization
Train on varied source distribution:

# Randomize source domain characteristics
augmented_source = apply_random_transforms(source, {
    "color": True,
    "texture": True,
    "lighting": True
})
# Helps generalize to unseen target domains

Evaluation
| Metric | Description |
|--------|-------------|
| Target accuracy | Performance on target |
| Source accuracy | Maintain source performance |
| Domain gap | Measure distribution difference |

Applications
| Domain | Example |
|--------|---------|
| Vision | Synthetic to real images |
| NLP | Formal to informal text |
| Medical | Hospital A to Hospital B |
| Robotics | Simulation to real robot |

Best Practices
- Analyze source-target distribution gap
- Start with simpler methods (finetuning)
- Use validation split from target domain
- Consider multiple source domains