Masked Image Modeling

The Problem

Self-supervised pretraining for language models is straightforward: BERT masks a few tokens and predicts them; GPT predicts the next token. Both work because language has inherent token-level structure. But how do you do the same for images? Pixels aren’t tokens. CNNs and ViTs need a different self-supervised signal. Pre-2021 vision SSL relied on contrastive methods (SimCLR, MoCo, DINO) that need careful augmentation pipelines and large batch sizes. The question: is there a vision analogue of BERT’s masked-language modeling — simple, scalable, and effective?

The Key Insight

Tokenize the image into patches (à la Vision Transformer), randomly mask a large fraction of patches (typically 75%), and ask the model to reconstruct the masked patches from the visible ones. The encoder learns to extract rich spatial features because reconstruction requires it. Unlike contrastive methods, MIM doesn’t need negatives, augmentation pipelines, or large batches — just the masking objective.

Mechanism in Plain English

1. Patchify: split the image into a grid of patches (e.g., 14x14 = 196 patches at 16x16 px each, for 224x224 input).
2. Randomly mask ~75% of patches. The encoder only sees the visible 25%.
3. Encode visible patches with a ViT. Output: per-patch embeddings (only for visible).
4. Decode: pass the encoded visible tokens + learned mask tokens for the masked positions through a small decoder. Predict the pixel values (or HOG features, or a discrete code) of the masked patches.
5. Loss: mean-squared error on the masked patches’ pixel values.
6. Discard the decoder after pretraining; the encoder is the foundation model.

ASCII Diagram

INPUT IMAGE (224 x 224)
       |
   [16 x 16 patchify]
       |
       v
   196 patches
       |
   [random mask: drop 147 (75%)]
       |
       v
   49 visible patches
       |
   [add positional embeddings]
       |
   [ViT-Large encoder (the to-be-foundation-model)]
       |
       v
   49 encoded tokens
       |
   [insert 147 mask tokens at original positions, all positions present]
       |
   [small ViT decoder]
       |
       v
   Reconstructed pixel values for all 196 patches
       |
   [MSE loss only on the masked 147 patches]

Math with Translation

The reconstruction loss:

$\mathcal{L}=\frac{1}{|M|}{\sum }_{i\in M}\Vert {\hat{p}}_i-p_i{\Vert }_2^2$

• $M$ = set of masked patch indices.
• ${\hat{p}}_i$ = the decoder’s predicted pixel values for patch $i$.
• $p_i$ = the original pixel values of patch $i$.
• Loss is normalized by the number of masked patches.

The encoder is forced to capture enough information about visible patches that the decoder can reconstruct missing ones. After pretraining, the encoder produces general-purpose features for downstream classification, segmentation, etc.

Concrete Walkthrough

MAE TRAINING ON IMAGENET:

  Image: 224x224x3 photograph.
  Patch: 16x16 -> 196 patches per image.
  Mask: random 75% (147 patches) -> 49 visible.
  
  Encoder ViT-Huge (632M params): processes only 49 visible tokens.
                                  ~3x faster than full-image processing.
  Decoder: 8-layer ViT, 512-dim. Takes 196 tokens (49 visible + 147 mask).
  
  Loss: MSE on 147 masked patches' pixels.

DOWNSTREAM TRANSFER:
  Take pretrained encoder. Discard decoder.
  Add classification head. Fine-tune on ImageNet labels.
  
  ImageNet top-1 accuracy: 87.8% (ViT-Huge MAE) vs 79.2% (ViT-Huge supervised).
  6+ point improvement over supervised baseline at the same model size.

SatMAE EXTENSION (for satellite imagery):
  - Mask across time as well as space.
  - Mask whole spectral groups, not just RGB patches.
  - Reconstruct multi-band, multi-temporal patches.
  - Same loss form, richer mask design.

What’s Clever

The first clever recognition: a high mask ratio (75%) is essential. Earlier vision SSL (BEiT, iBOT) used lower mask ratios (15%, like BERT). MAE shows that 75% is the sweet spot for images — at 15%, the task is too easy (lots of redundancy nearby); at 90%, too hard (no context). The 75% number reflects the spatial redundancy of natural images.

The second clever move: asymmetric encoder-decoder. The encoder only processes the 25% visible patches — saving compute by 4x. The decoder is small (a fraction of the encoder size) because its job is just reconstruction, not feature extraction. After pretraining, the decoder is thrown away. This asymmetry is what makes MAE 3-4x faster than contrastive methods at similar quality.

The third recognition: MIM works at scale and doesn’t need augmentation. Contrastive methods need elaborate augmentation pipelines (random crop, color jitter, gaussian blur, etc.) to define what counts as the “same” image. MIM needs only the mask — the augmentation is the masking itself. This makes MIM much simpler to apply to new domains (medical imaging, satellite imagery, video) where the right augmentations aren’t obvious.

The fourth recognition: reconstructing low-level pixels still produces high-level features. Surprising but verified empirically: the encoder doesn’t get distracted by the pixel-prediction objective; it learns features that transfer to high-level tasks (classification, segmentation, detection). The intuition: to reconstruct, the encoder needs to know what’s in the image; to know what’s in the image, it learns object-level structure.

Key Sources

• mae-masked-autoencoders-scalable-vision-learners — the foundational paper
• satmae-pretraining-transformers-temporal-multispectral-satellite-imagery — adaptation to satellite imagery
• dino-self-supervised-vision-transformers — alternative SSL paradigm (contrastive); useful for comparison

Related Concepts

• self-supervised-learning — MIM is a major branch
• vision-transformer — the standard backbone
• patch-embeddings — what gets masked
• foundation-models — MIM is the dominant pretraining recipe for vision FMs
• geospatial-foundation-models — SatMAE etc. extend MIM to satellite domain

Open Questions

• Targets beyond pixels: BEiT-v2 reconstructs discrete tokens; HOG-MAE reconstructs HOG features. Which target is best? Pixels seem to work for natural images; semantic targets help for some downstream tasks.
• Mask design: random vs structured (block masking, token-level smart masking). MAE uses random; some work suggests structured masks help.
• Combination with contrastive: hybrid methods (DINOv2, iBOT) combine MIM with contrastive losses. When does this help?
• Cross-modal MIM: how to mask in image-text or image-audio pairs? VideoMAE for video, AudioMAE for audio. The pattern generalizes.
• Inference cost: pretrained MAE encoders are ViT-Huge or larger. Distillation to smaller models is an active area.