‘Transformer’

Overview of our proposed pre-training framework.
Input CT images are randomly cropped into sub-volumes and
augmented with random inner cutout and rotation, then fed to the
Swin UNETR encoder as input. We use masked volume inpainting,
contrastive learning and rotation prediction as proxy tasks for
learning contextual representations of input images

Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

Jul. 25, 2022—Tang, Yucheng, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. “Self-supervised pre-training of swin transformers for 3d medical image analysis.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20730-20740. 2022. Full text: Abstract Vision Transformers (ViT)s have shown great performance in self-supervised...