ML Wiki

Transfer Learning

6 min read

What It Is

Transfer learning is the practice of pre-training a model on a large general dataset, then adapting it to a smaller, task-specific one — reusing learned representations instead of training from scratch. The core bet: features learned on a broad pretraining distribution (natural language, internet images) capture structure that transfers to downstream tasks, requiring far less task-specific data and compute.

Why It Matters

Transfer learning is the dominant paradigm in modern ML. Without it, every task would require training from scratch on millions of examples — prohibitively expensive for most practitioners. The pre-train-then-fine-tune paradigm concentrates expensive compute into one large pretraining run, then amortizes that cost across countless downstream applications. GPT-4, CLIP, and ViT are all foundation models — trained once at massive scale, then transferred to thousands of different tasks by fine-tuning.

The economic consequence is severe: transfer learning is what makes frontier AI accessible to organizations that can’t train foundation models themselves. Fine-tuning a 7B model takes hours on one GPU; pretraining it from scratch takes months on thousands.

How It Works

The Three-Stage View

Stage 1 — Pretraining:
  Large, diverse dataset (Common Crawl, JFT-300M, LAION-5B)
  Self-supervised or weakly supervised objective
  Compute: thousands of GPU-days to millions
  Result: a foundation model with general representations

Stage 2 — Task Adaptation:
  Replace task-specific head (e.g., swap language model head for
  classification head)
  Optionally freeze pretrained weights (linear probing) or
  allow full weight updates (full fine-tuning)

Stage 3 — Fine-Tuning:
  Train on task dataset (100 to 1M examples, depending on task)
  Low learning rate (10-100× smaller than pretraining)
  Few epochs (1-5 typical, vs. many for pretraining)
  Result: task-specialized model inheriting pretrained representations

What Transfers and Why

Transfer learning works when the pretraining task teaches features useful for the downstream task. Two mechanisms:

Feature reuse: Low-level features (edges, textures in vision; syntax, word co-occurrences in NLP) learned during pretraining are directly useful for downstream tasks. These transfer even if the output task is completely different.

Representational capacity: Pretraining builds compressed representations of the data distribution. A well-pretrained LLM encodes semantic structure that transfers to classification, summarization, and generation — even though pretraining used only next-token prediction.

Linear Probing vs. Full Fine-Tuning

Two adaptation extremes:

Linear probing: Freeze all pretrained weights. Add only a linear classification head. Train only the head.

  • Best for: evaluating representation quality, limited data, preventing catastrophic forgetting
  • Worst for: tasks with significant distribution shift from pretraining

Full fine-tuning: Update all weights with a small learning rate.

  • Best for: tasks with enough data (>10K examples), where task distribution differs from pretraining
  • Risk: catastrophic forgetting if learning rate is too high or data too limited

LoRA / PEFT (middle ground): Keep pretrained weights frozen; add small trainable low-rank adapters to attention matrices. Combines fine-tuned performance with catastrophic forgetting prevention. The dominant approach for LLM fine-tuning.

ViT: Transfer Learning Over Inductive Bias

ViT’s result is the most striking demonstration of transfer learning’s power. CNNs have strong inductive bias — convolutions enforce local processing and translation equivariance, which happen to be correct for natural images. ViT has no such bias: it processes image patches with global attention, potentially attending across the entire image from layer 1.

On ImageNet alone (1.28M images): ViT underperforms CNNs because it lacks the inductive bias to generalize from limited data. On JFT-300M (300M images): ViT outperforms CNNs because sufficient data lets it learn locality and translation structure from scratch.

Transfer learning is the mechanism: train at enormous scale to compensate for absent inductive bias, then transfer to the downstream task. The lesson generalizes: when data is abundant, learning structure beats hard-coding it.

Data Efficiency of Transfer

Task: classifying 1,000 ImageNet categories

Training from scratch:       need 1.28M images for competitive accuracy
Linear probing (ViT/JFT):    need ~50K images for competitive accuracy
Full fine-tuning (ViT/JFT):  need ~10K images for competitive accuracy
CLIP zero-shot:              need 0 images (zero-shot transfer)

Each stage of transfer — from pretraining better representations to more task-aligned pretraining to providing demonstrations — reduces the downstream data requirement by 10-100×.

What’s Clever

The non-obvious insight: the pretraining task doesn’t need to match the downstream task at all, as long as the pretraining task requires learning useful features. BERT is pretrained on masked language modeling (predict a randomly hidden word) — a task with no immediate application. But to predict a hidden word well, the model must learn syntax, semantics, coreference, and world knowledge. All of these transfer to downstream tasks.

Similarly, CLIP is pretrained on image-text matching — never directly trained on ImageNet — but implicitly learns visual categories because the natural language captions name them. The pretraining task is a proxy that forces learning useful representations.

The key failure mode: negative transfer. If the pretraining distribution is too far from the downstream task, the pretrained weights are not just neutral — they’re actively harmful because fine-tuning starts from the wrong point. This happens when adapting a model trained on English text to medical code, or a model trained on natural photos to satellite imagery with very different statistics.

Key Sources

  • vision-transformer — ViT’s performance profile is only understandable through the lens of transfer learning
  • inductive-bias — transfer learning can substitute for architectural inductive bias given sufficient pretraining data
  • zero-shot-transfer — the extreme end of transfer: no task-specific adaptation at all
  • lora — efficient fine-tuning that enables transfer while preserving pretrained representations
  • sft — SFT is a form of transfer learning applied to language model alignment
  • distillation — an alternative to transfer learning for model compression; learns from a larger pretrained teacher

Open Questions

  • What determines the effective transfer distance — how different can source and target tasks be before transfer becomes negative?
  • Does more pretraining data always improve downstream transfer, or is there a saturation point?
  • Can multi-task pretraining (simultaneous training on diverse tasks) achieve better transfer than single-objective pretraining?
  • How do we measure what has transferred vs. what was learned fresh during fine-tuning?

Graph View

Backlinks