Swin Transformer: Hierarchical Vision with Shifted Windows

ViT’s Limitation

ViT computes full self-attention across all 196 patches. This is O(n²) in the number of patches — manageable at 224×224 but breaks down for high-resolution images (e.g., 1024×1024 for detection).

CNNs build hierarchical feature maps: early layers capture fine details (many small feature maps), later layers capture coarse semantics (fewer, larger feature maps). ViT has no such hierarchy.

Swin Transformer fixes both problems.

Two Key Ideas

1. Window-Based Attention (W-MSA)

Instead of attending over the whole image, Swin divides the patch grid into non-overlapping local windows of M×M patches (M=7 by default).

Self-attention runs within each window independently. If the image has n patches and windows have M² patches, complexity drops from O(n²) to O(n·M²) — linear in image size.

2. Shifted Windows (SW-MSA)

Window-based attention is efficient but windows are isolated: a patch at the right edge of window 1 never interacts with its neighbour at the left edge of window 2.

The shift trick: alternate between regular and shifted window configurations every layer. Shifted windows cross the old boundaries, allowing information to flow across the grid.

To handle patches at the edges that don’t fill a full window, cyclic shift and a masking strategy handle the boundary conditions efficiently.

Hierarchical Feature Maps

After each stage, patch merging concatenates 2×2 neighbouring patches and projects them to 2×d dimensions. This halves spatial resolution and doubles channel width — mimicking CNN downsampling.

These multi-scale features plug directly into standard detection heads (FPN, DETR) and segmentation decoders — something ViT cannot easily do.

Where Swin Wins

Swin won COCO object detection and ADE20K segmentation upon release. Its hierarchical design and local attention make it the preferred ViT variant for dense prediction tasks.