Research Short Computer Vision and Pattern Recognition (cs.CV)

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

Microsoft Research Asia

↑ Uploaded by SheQAI ✓ Grounding-verified by TeX2Vid Grounding report · 7 claims traced Code ↗ ▶ 17

⧉ Cite

APA

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (n.d.). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

BibTeX

@article{liu,
  title  = {Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},
  author = {Ze Liu and Yutong Lin and Yue Cao and Han Hu and Yixuan Wei and Zheng Zhang and Stephen Lin and Baining Guo},
}

📄 Transcript

Vision needs a Transformer backbone.

Transformers dominate language, but adapting them to vision

is hard. Visual entities vary enormously in scale,

unlike fixed-scale word tokens, and images pack far

more pixels than sentences pack words.

The key idea is a hierarchical backbone that

computes self-attention within non-overlapping local windows, giving linear

complexity in image size. To connect windows, a

shifted window scheme alternates the partitioning between consecutive

blocks, adding cross-window links without the memory cost

of sliding windows. Patch merging builds feature maps

at multiple scales.

The result: 58.7 box AP and 51.1 mask

AP on COCO, plus 87.3% top-1 on ImageNet,

surpassing prior state of the art by a

large margin.

Read the paper for details.

Abstract

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at https://github.com/microsoft/Swin-Transformer.

✓ Claims & sources (7)

Each claim in this video, with the span of the paper it comes from.

Key point Vision Transformers face large scale variation and high pixel resolution unlike fixed-scale word tokens.

Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text.

Key point Standard self-attention has quadratic complexity with image size, making dense prediction intractable.

Second, image pixels have much higher resolution than words in text, and the self-attention computation in standard vision Transformers has quadratic computational complexity with respect to image size, making dense prediction tasks like semantic segmentation intractable on high-resolution images.

Key point Self-attention runs inside non-overlapping local windows, giving linear complexity in image size.

The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection.

Key point Shifted windows alternate partitions across blocks to add cross-window connections efficiently.

To introduce cross-window connections while preserving efficiency, a shifted window (SW-MSA) partitioning alternates between regular window partitioning (W-MSA) and a configuration displaced by half the window size in consecutive blocks.

Key point Reaches 58.7 box AP and 51.1 mask AP on COCO, surpassing prior state of the art.

Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones.

Key point Hierarchical feature maps built via patch merging enable use as a general-purpose backbone.

The architecture builds hierarchical feature maps across four stages by merging neighboring patches via patch merging layers.

Key point Achieves 87.3 top-1 accuracy on ImageNet-1K and 53.5 mIoU on ADE20K val.

including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val).

Comments (0)

No comments yet.

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related videos